# Improving Software Defect Prediction in Noisy Imbalanced Datasets
## 3 Materials and Methods
### Predictor Construction
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Projects** & **Versions** & **Total Instances** & **Defective Instances** & **Unbalance Ratio \(r\)** \\ \hline ant & 1.3, 1.4, 1.5, 1.6 & 947 & 184 & 4.147 \\ camel & 1.0, 1.2, 1.4, 1.6 & 2784 & 562 & 3.954 \\ poi & 2 & 314 & 37 & 7.486 \\ synapse & 1.0, 1.1, 1.2 & 635 & 162 & 2.920 \\ log4j & 1 & 138 & 34 & 2.971 \\ jedit & 3.2, 4.0, 4.1, 4.2, 4.3 & 1749 & 303 & 4.772 \\ PDE & 1 & 1497 & 209 & 6.163 \\ JDT & 1 & 997 & 206 & 3.840 \\ velocity & 1.6 & 229 & 78 & 1.936 \\ xerces & 1.2, 1.3 & 893 & 140 & 5.379 \\ mylyn & 1 & 1862 & 245 & 6.600 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Datasets. SMOTE-RSB [53], SMOTE-FRST-2T [54], SMOTE [20], SMOTE-Tomek Links [42], Borderline-SMOTE [43], ADASYN [44], MSMOTE [50], Gaussian-SMOTE [47], and CCR [52]. Scikit-learn and the open-source library \"smote_variants\" [60] on GitHub were used to implement these methods. ## 5 Results

### Answer to RQ1: Is US-PONR Effective in SDP Datasets? In order to answer RQ1, three small experiments were conducted:

1. Determining the optimization parameter. 2. Comparing the results of US-PONR with the results of just US (undersampling) or PONR alone. 3. Comparing US-PONR with the SOTA data pre-processing method. The first experiment determined the optimal parameter of each dataset. These parameters were then used for the other two experiments. The second experiment sought to prove the necessity of using both US and PONR in the proposed method. The third experiment sought to validate the effectiveness of US-PONR. The dataset used in these experiments was the original PROMISE dataset without any introduced noise (shown in Table 5). #### 5.1.1 Determining the Optimization Parameter

To obtain the optimal parameter, different steps were taken to process undersampling on the selected datasets; step size was set at 1 between [1, r], where r is the degree of imbalance in the dataset. We trained the CV-based aggregation model under different conditions and then compared with the average AUC obtained by cross-validation; the undersampling ratio with the largest average AUC was selected as the optimal parameter. When conducting this experiment, we discovered that the search interval was usually too large, so we recommend that one first use step = 1 to search for the interval where there may be optimal parameters and then use step = 0.2 to find the optimal parameters. Our test results indicate that the value interval should be set as follows: ant [3, 4], camel [3, 39], poi [5, 6], JDT [2.6, 3.4], synapse [2, 3], jedit [3.6, 4.4], velocity [1, 1.8], xerces [4.4, 5.2], log4] [1.6, 2.4], PDE [4.6, 5.4], and mylyn [4.6, 5.4]. The results for parameter optimization are shown in Table 6:

These results show that the optimal parameters of each dataset are as follows: ant (3.8), camel (3.9), JDT (3.2), jedit (4.2), log4j (2.2), mylyn (5.4), PDE (5), poi (5.8), synapse (2.4), velocity (1.8), and xerces (4.8). \begin{table}
\begin{tabular}{c c c} \hline \hline  & **US-PONR** & \\  & **rus** & **AUC** \\ \hline ant & 3.8 & 0.8218 \\
**camel** & 3.9 & 0.8612 \\
**JDT** & 3.4 & 0.9257 \\
**jedit** & 4.4 & 0.9381 \\
**log4j** & 2.2 & 0.8716 \\
**mylyn** & 5.4 & 0.9116 \\
**PDE** & 5 & 0.9292 \\
**poi** & 5.8 & 0.9416 \\
**synapse** & 2.4 & 0.7811 \\
**velocity** & 1.9 & 0.7842 \\
**xerces** & 4.8 & 0.9442 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Parameter-undersampling_ratio optimization. #### 5.1.2 Comparing US-PONR with US and PONR

After the best parameters were obtained for each dataset, the dataset pre-processed with US-PONR or only US or only PONR was trained with a CV-based aggregation model, individually, and the AUC for each of the three methods was compared as the output result. Figure 2 shows how the AUC of US-PONR under the optimal parameters compared with the AUC of US and PONR, individually. US-PONR performed better than both US and PONR in most datasets but performed less well than PONR alone in the ant, log4j, and synapse datasets.