# Causally Remove Negative Confound Effects of Size Metric for Software Defect Prediction
## 4 Experiments and Results
### Results  
Through extensive experiments, we empirically verified the effectiveness of the proposed CNCERM. Tables 6-13 show the experimental results, concluding precision rate, recall rate, F1 score, and improved F1 score. Figures 3-5 intuitively present each model's performance. We have the following observations and analysis:  
1. Our proposed CNCERM can effectively improve the performance of the LR model. Compared with that of LR, the F1 value of CNCERM+LR improved by 4.1% in CM1, 4.0% in KC1, 3.3% in KC3, 5.2% in MC1, 1.5% in MC2, 1.3% in MW1, 5.0% in PC3, and 2.1% in PC4, with an average of 3.3%. Compared with the best performance of LR and LCERM + LR (indicated by * in Tables 6-13), the F1 value of CNCERM + LR increases by 4.1% in CM1, 2.3% in KC1, 3.3% in KC3, 5.2% in MC1, 1.5% in MC2, 1.3% in MW1, 5.0% in PC3, 2.1% in PC4, with an average 3.1%. Among all three prediction models with LR as the classifier, CNCERM + LR performs best. 2. CNCERM + LR performs better than LR, shown in Figure 3, which verifies that the confounding effects of the size metric do affect the prediction performance of LR. The proposed CNCERM can effectively quantify the confounding effects of the class metric and then analyze the direction of the confounding effects. After removing the negative confounding effects, LR significantly improves. 3. LCERM + LR performs worse than LR in general, as shown in Figure 3, which means that the traditional confounding removal method is unsuitable for software defect prediction and an inappropriate removal method cannot improve the LR's performance. Moreover, removing the confounding effects of size metric is not necessarily beneficial to software defect prediction. We should only remove the negative confounding effects that are not conducive to defect prediction. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Model** & **Precision** & **Recall** & **F1** & **Impr F1** \\ \hline \multirow{4}{*}{MC2} & LR & 51.4\% & 49.6\% & 49.9\% * & \multirow{4}{*}{1.5\%} \\  & LCERM + LR & 57.3\% & 31.8\% & 39.9\% & \\ \cline{1-1}  & NN & 31.9\% & 47.9\% & 36.5\% & \\ \cline{1-1}  & CRNCEM + LR & 51.9\% & 51.8\% & 51.4\% & \\ \hline \hline \end{tabular}  
* indicates the best performance of the compared models.  
\end{table} 
Table 10: Results of MC2. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Model** & **Precision** & **Recall** & **F1** & **Impr F1** \\ \hline \multirow{4}{*}{PC3} & LR & 48.0\% & 16.4\% & 23.6\% * & \multirow{4}{*}{5.0\%} \\  & LCERM + LR & 46.3\% & 16.3\% & 23.6\%* & \\ \cline{1-1}  & NN & 12.4\% & 50.1\% & 19.5\% & \\ \cline{1-1}  & CRNCEM + LR & 48.7\% & 21.0\% & 28.6\% & \\ \hline \hline \end{tabular}  
* indicates the best performance of the compared models.  
\end{table} 
Table 12: Results of PC3. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset** & **Model** & **Precision** & **Recall** & **F1** & **Impr F1** \\ \hline \multirow{4}{*}{MC2} & LR & 51.4\% & 49.6\% & 49.9\% * & \multirow{4}{*}{1.5\%} \\  & LCERM + LR & 57.3\% & 31.8\% & 39.9\% & \\ \cline{1-1}  & NN & 31.9\% & 47.9\% & 36.5\% & \\ \cline{1-1}  & CRNCEM + LR & 51.9\% & 51.8\% & 51.4\% & \\ \hline \hline \end{tabular}  
* indicates the best performance of the compared models.  
\end{table} 
Table 10: Results of MC2. 4. Figure 3 intuitively shows that CNCERM + LR performs better than LCERM + LR, indicating that our proposed CNCERM is more suitable for software defect prediction than the linear-based confounding removal method. The predictive ability of LCERM+LR is worse than that of LR, and the predictive ability of CNCERM + LR is stronger than that of LR. The existing confounding removal methods are not suitable for software defect prediction, which further illustrates the necessity of this article. 5. Figure 4 intuitively shows that CNCERM + LR and LR have similar precision rates, and each has wins and losses but not much difference. Figure 5 shows that CNCERM + LR has slightly better recall performance than that of LR.