## 5 Results and Analyses
### Prediction Models  
In this section, the results of the empirical study are presented and analyzed. Four prediction models involving three individual models (i.e., NB, SVM, and KNN) and one ensemble model (RF) were employed on seven datasets. Each prediction model was constructed using four different datasets extracted from the four scenarios analyzed (see Table 1). Therefore, the total number of prediction models was 112:7 datasets \(\times\) 4 scenarios \(\times\) 4 prediction models. Figure 3 shows the box plots of the AUC of each prediction model across the seven datasets for the four scenarios (see Table 2). The mean value of the AUC values is indicated by an \"X\", the upper and lower lines of the box represent the first and third quartile, and the middle horizontal line across the box represents the middle quartile. The prediction model \begin{table}
\begin{tabular}{l l l l l l l l l l l l} \hline \hline  & \multicolumn{3}{c}{**Before Applying SMOTE**} & \multicolumn{3}{c}{**\% of SMOTE**} & \multicolumn{3}{c}{**After Applying SMOTE**} \\ \hline
**Dataset** & **\# Classes** & **\# True** & **\% True** & **\# False** & **\% False** & **\% False** & **\# Classes** & **\# True** & **\% True** & **\% False** & **\% False** & **\% False** & **\% False** \\ \hline antirt4 & 459 & 46 & 10.02 & 413 & 89.98 & 142 & 46 & 32.39 & 96 & 67.61 \\ \hline junit & 675 & 27 & 4.00 & 648 & 96.00 & 83 & 27 & 32.53 & 56 & _67.47_ \\ \hline MapDB & 451 & 16 & 3.55 & 435 & 96.45 & 49 & 16 & 32.65 & 33 & 67.35 \\ \hline mcMMO & 309 & 12 & 3.88 & 297 & 96.12 & 37 & 12 & 32.43 & 25 & _67.57_ \\ \hline mct & 2207 & 60 & 2.72 & 2147 & 97.28 & 186 & 60 & 32.26 & 126 & 67.74 \\ \hline \hline oryx & 551 & 30 & 5.44 & 521 & 94.56 & 93 & 30 & 32.26 & 63 & 67.74 \\ \hline titan & 1525 & 52 & 3.41 & 1473 & 96.59 & 161 & 52 & 32.30 & 109 & 67.70 \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Scenarios in the empirical study. \begin{table}
\begin{tabular}{l l l l l l l l l l l l} \hline \hline  & \multicolumn{3}{c}{**Before Applying Spread Sub-Sample**} & \multicolumn{3}{c}{**After Applying Spread Sub-Sample**} \\ \hline
**Dataset** & **\# Classes** & **\# True** & **\% True** & **\% True** & **\# False** & **\% False** & **\#** & **\# True** & **\% True** & **\% True** & **\% False** & **\% False** & **\% False** \\ \hline antirt4 & 459 & 46 & 10.02 & 413 & 89.98 & 142 & 46 & 32.39 & 96 & 67.61 \\ \hline junit & 675 & 27 & 4.00 & 648 & 96.00 & 83 & 27 & 32.53 & 56 & _67.47_ \\ \hline MapDB & 451 & 16 & 3.55 & 435 & 96.45 & 49 & 16 & 32.65 & 33 & 67.35 \\ \hline mCMMO & 309 & 12 & 3.88 & 297 & 96.12 & 37 & 12 & 32.43 & 25 & _67.57_ \\ \hline mct & 2207 & 60 & 2.72 & 2147 & 97.28 & 186 & 60 & 32.26 & 126 & 67.74 \\ \hline oryx & 551 & 30 & 5.44 & 521 & 94.56 & 93 & 30 & 32.26 & 63 & 67.74 \\ \hline titan & 1525 & 52 & 3.41 & 1473 & 96.59 & 161 & 52 & 32.30 & 109 & 67.70 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Results before and after applying SMOTE. which has the highest \"X\" values and the spread of the box is considered to be preferable. It is important to mention that because of the way that the scenarios were constructed, the test sets used for evaluating sampling methods in the third and fourth scenarios were different from those used for the non-sampling methods in the first and second scenarios. For this reason, the comparison between these two distinct groups of scenarios (first and second versus third and fourth) has not been carried out as it would be invalid. It is observed in Figure 3 that across all four scenarios, RF attained the highest prediction accuracy. Figure 4 illustrates the AUC results obtained from each prediction model across seven datasets in four scenarios, in which the higher AUC value indicates the better result. The results of the third scenario provide valuable insights into the positive influence of ensemble sampling techniques to improve the prediction accuracy of prediction models.