## 5 Results and Analyses
### Prediction Models
#### 5.3.8 Discussion and Answers to Research Questions  
This section provides the discussion of the results presented above and answers the **RQs** for this study. RQ1. What is the impact of ensemble FS techniques on the performance of prediction models? \begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline
**Scenario** & **First** & \multicolumn{5}{c}{**Second**} & \multicolumn{5}{c}{**Third**} & \multicolumn{5}{c}{**Fourth**} \\ \hline  & **Mtry** & **AUC-T** & **AUC-D** & **Mtry** & **AUC-T** & **AUC-D** & **Mtry** & **AUC-T** & **AUC-D** & **Mtry** & **AUC-T** & **AUC-D** \\ \hline antlr4 & 3 & **0.81** & 0.7 & 2 & **0.84** & 0.81 & 6 & **0.94** & 0.89 & 1 & **0.87** & 0.85 \\ \hline junit & 15 & **0.84** & 0.73 & 3 & **0.81** & 0.75 & 5 & **0.95** & 0.92 & 5 & **0.89** & 0.85 \\ \hline MapDB & 2 & **0.90** & 0.79 & 10 & **0.72** & 0.7 & 8 & **0.96** & 0.89 & 2 & **0.98** & 0.95 \\ \hline mcMMO & 11 & **0.89** & 0.79 & 2 & **0.89** & 0.7 & 6 & **1.00** & 0.98 & 5 & **0.98** & 0.96 \\ \hline mct & 2 & **0.96** & 0.89 & 4 & **0.96** & 0.89 & 1 & **0.99** & 0.99 & 14 & **0.99** & 0.98 \\ \hline oryx & 8 & **0.82** & 0.81 & 1 & **0.85** & 0.8 & 6 & **0.94** & 0.92 & 14 & **0.90** & 0.85 \\ \hline titan & 8 & **0.90** & 0.81 & 6 & **0.85** & 0.8 & 15 & **0.99** & 0.97 & 2 & **0.93** & 0.9 \\ \hline
**Average** & NA & 0.87 & 0.79 & NA & 0.85 & 0.78 & NA & 0.97 & 0.94 & NA & 0.93 & 0.91 \\ \hline
**\% of change** & NA & 10.13\% & NA & 8.97\% & NA & 3.19\% & NA & 2.20\% & \\ \hline \hline \end{tabular}
\end{table} 
Table 17: Scenario AUC values for performance evaluation of RF with default and Mtry parameter tuning. Figure 5: Multiple comparisons for prediction models using AUC. Ensemble FS techniques in the second scenario improved the prediction accuracy of NB and KNN models. Analyzing the likely reasons behind this finding, firstly, NB is called naive because it creates conditional assumptions, which are independent from the features [37]. Second, using the Euclidean measure,  
KNN determines the closest neighbors, which are also independent from the features [40]. Therefore, these models may have performed well with FS techniques because they do not perform attribute selection [45]. However, ensemble FS techniques had no clear impact on the overall performance of SVM and RF. This result may be explained by the fact that SVM algorithm includes the C parameter that selects the number of features, and the kernel function creates a suitable feature space [32]. Regarding the RF result, this occurs because RF has multiple decision trees that apply the same concept of FS using a top-down greedy search algorithm to choose the best feature at each step [58]. Therefore, SVM and RF algorithms already perform the FS concept during their creation. Additionally, RF includes many decision trees, which perform adequately with an imbalanced dataset because they tend to build several tests to recognize the difference between the minority and majority classes [59]. For this reason, RF achieved the best prediction accuracy in the second scenario. RQ2. What is the impact of ensemble sampling techniques on the performance of prediction models? The machine learning models with ensemble sampling techniques achieved good performance. Again, RF achieved the best results of AUC values across all the datasets except mcMMO. These findings may help us to conclude that performing ensemble sampling techniques and removing inappropriate features in the data analysis step without applying ensemble FS is sufficient to predict change proneness accurately. This is consistent with what has been found in a series of recent studies [8; 9] that used several types of sampling techniques: SMOTE [8], UPSAMPLE, SMOTE and RUSBoost [9], and emphasizes the effect of these techniques to improve the performance of the prediction models. Interestingly, ensemble sampling techniques in this particular case increased the prediction accuracy of SVM and RF.