## 5 Results
### Impact of the Sampling Methods on SVM and Random Forest  
In this experiment, the objective is to identify the most supportive sampling algorithm for the selected classifiers. The parameter setting for each sampler is the following:  
* SMOTE: _sampling_strategy_ ='minority', _k_neighbors_ = 5, and _ratio_ = None.
* Borderline SMOTE: _sampling_strategy_ ='minority', _k_neighbors_ = 5, _m_neighbors_ = 10, and _kind_ = 'borderline-1'.
* ADASYN: _sampling_strategy_ ='minority', _n_neighbors_ = 5, and _ratio_ = None.
* ROS: _sampling_strategy_ ='minority', and _ratio_ = None.
* RUS: _sampling_strategy_ ='majority', _replacement_ = False, and _ratio_ = None.  
Table 3 presents the results that were obtained by the two classifiers when enriched with sampling algorithms.  
Under this experimental setup and for the case study, we may reach the following conclusions after analyzing Table 3:  
* RUS was the best sampling approach in the scenario where the presence of false negatives is a critical concern (recall).
* We do not recommend resampling if false positives entail dire consequences (precision).
* In terms of \(F1\)-score, SMOTE and Borderline SMOTE mainly performed well.
* ROS seems to be the leading choice when considering ROC-AUC alone.
* Finally, an apparent inconsistency in the impact of sampling methods occurs when PR-AUC is considered decisive. On the one hand, SVM achieved the best results when ROS was applied (followed by RUS). On the other hand, contrary to expectations, no sampling algorithm improved the performance of Random Forest.  
For these reasons, we have decided to classify via SVM and resample through ROS. These algorithms jointly offered the best compromise among PR-AUC, ROC-AUC, and sensitivity (it is  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Classifier**} & \multirow{2}{*}{**Sampling Algorithm**} & \multicolumn{4}{c}{**Threshold Metrics**} & \multicolumn{2}{c}{**Ranking Metrics**} \\ \cline{3-7}  & & **Recall** & **Precision** & **\(F\)-Measure** & **ROC-AUC** & **PR-AUC** \\ \hline \multirow{8}{*}{SVM} & None (Base version) & 0.5658 & 0.9456 \({}^{\text{\textminus}}\)S & 0.7069 & 0.9597 & 0.8545 \\  & SMOTE & 0.8278 & 0.7511 & 0.7873 \({}^{\text{\textminus}}\) & 0.9681 & 0.8485 \\  & Borderline SMOTE & 0.8195 & 0.7671 & 0.7921 \({}^{\text{\textminus}}\)S & 0.9661 & 0.8515 \\  & ADASYN & 0.8332 & 0.7376 & 0.7821 & 0.9687 \({}^{\text{\textminus}}\) & 0.8490 \\  & ROS & 0.8585 & 0.7217 & 0.7828 & 0.9693 \({}^{\text{\textminus}}\)S & 0.8561 \({}^{\text{\textminus}}\)S \\  & RUS & 0.8596 \({}^{\text{\textminus}}\)S & 0.7074 & 0.7752 & 0.9693 \({}^{\text{\textminus}}\)S & 0.8550 \\  & None (Base version) & 0.5765 & 0.9355 \({}^{\text{\textminus}}\)S & 0.7129 & 0.9627 & 0.8552 \({}^{\text{\textminus}}\)S \\  & SMOTE & 0.7114 & 0.8820 & 0.7872 \({}^{\text{\textminus}}\)S & 0.9639 & 0.8481 \\ Random Forest & Borderline SMOTE & 0.7041 & 0.8917 & 0.7865 & 0.9624 & 0.8503 \\  & ADASYN & 0.7151 & 0.8749 & 0.7866 & 0.9633 & 0.8495 \\  & ROS & 0.6759 & 0.9050 & 0.7730 & 0.9647 \({}^{\text{\textminus}}\)S & 0.8429 \\  & RUS & 0.8641 \({}^{\text{\textminus}}\)S & 0.5523 & 0.6736 & 0.9621 & 0.8428 \\ \hline \hline \end{tabular}  
* the highest value of the metric; \({}^{\text{\textminus}}\) the second-highest value of the metric; \({}^{\text{\textminus}}\) the highest value of the metric for a specific classifier.  
\end{table}
Table 3: Average of the metrics calculated through 10-fold cross-validation for both classifiers with sampling.  
simultaneously the best one in PR-AUC and ROC-AUC and the third-best in recall). SVM improved its performance by applying ROS prior to training; its metrics were increased in the following manner:  
* Recall: from 0.5658 to 0.8558.
* \(F\)-measure: from 0.7069 to 0.7828.
* ROC-AUC: from 0.9597 to 0.9693.
* PR-AUC: from 0.8545 to 0.8561.  
Appendix B contains the plots of the ROC and PR curves, fold-by-fold, for each pair of algorithms in Table 3, which can be checked for further details.