## 4 Experiments
### Implementation and Evaluation
\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & **ACC** & **PR** & **Recall** & **Weighted F1** & **Macro F1** & **MCC** & **FMI** \\ \hline \hline
**Manual Augmentation Only** & & & & & & & \\ \hline Decision Tree & 95.66 & 0.94 & 0.95 & 0.93 & 0.90 & 0.93 & 0.94 \\ \hline Random Forest & 91.66 & 0.85 & 0.91 & 0.88 & 0.81 & 0.87 & 0.88 \\ \hline AdaBoosted DT & 95.00 & 0.93 & 0.95 & 0.93 & 0.91 & 0.93 & 0.91 \\ \hline OLA & 89.00 & 0.82 & 0.89 & 0.85 & 0.77 & 0.84 & 0.87 \\ \hline LCA & 83.99 & 0.75 & 0.84 & 0.78 & 0.69 & 0.76 & 0.78 \\ \hline KNORA-U & 88.00 & 0.81 & 0.88 & 0.83 & 0.76 & 0.82 & 0.84 \\ \hline KNORA-E & 91.66 & 0.87 & 0.91 & 0.89 & 0.82 & 0.87 & 0.88 \\ \hline META-DES & 90.00 & 0.83 & 0.90 & 0.86 & 0.78 & 0.85 & 0.87 \\ \hline
**Manual Augmentation + Augmentation with SMOTE** & & & & & & \\ \hline Decision Tree & 94.98 & 0.92 & 0.94 & 0.93 & 0.90 & 0.92 & 0.91 \\ \hline Random Forest & 90.00 & 0.85 & 0.90 & 0.86 & 0.80 & 0.85 & 0.84 \\ \hline AdaBoosted DT & 96.66 & 0.94 & 0.96 & 0.95 & 0.93 & 0.95 & 0.95 \\ \hline OLA & 92.66 & 0.92 & 0.92 & 0.91 & 0.85 & 0.89 & 0.90 \\ \hline LCA & 91.33 & 0.89 & 0.91 & 0.89 & 0.85 & 0.87 & 0.85 \\ \hline KNORA-U & 92.66 & 0.92 & 0.92 & 0.91 & 0.86 & 0.86 & 0.89 \\ \hline KNORA-E & 90.00 & 0.88 & 0.90 & 0.88 & 0.80 & 0.89 & 0.85 \\ \hline META-DES & 92.66 & 0.90 & 0.92 & 0.91 & 0.85 & 0.89 & 0.90 \\ \hline
**Manual Augmentation + Augmentation with Borderline-SMOTE** & & & & & & \\ \hline Decision Tree & 96.00 & 0.93 & 0.94 & 0.94 & 0.92 & 0.94 & 0.93 \\ \hline Random Forest & 92.66 & 0.89 & 0.92 & 0.90 & 0.85 & 0.89 & 0.88 \\ \hline AdaBoosted DT & 96.66 & 0.95 & 0.96 & 0.95 & 0.93 & 0.95 & 0.94 \\ \hline OLA & 90.00 & 0.88 & 0.90 & 0.88 & 0.82 & 0.85 & 0.83 \\ \hline LCA & 90.66 & 0.88 & 0.90 & 0.89 & 0.83 & 0.87 & 0.85 \\ \hline KNORA-U & 92.33 & 0.91 & 0.92 & 0.91 & 0.85 & 0.89 & 0.89 \\ \hline KNORA-E & 92.66 & 0.91 & 0.92 & 0.91 & 0.85 & 0.89 & 0.89 \\ \hline META-DES & 92.66 & 0.90 & 0.92 & 0.90 & 0.84 & 0.89 & 0.90 \\ \hline
**Manual Augmentation + Augmentation with Borderline-SMOTE** & & & & & & \\ \hline Decision Tree & 92.66 & 0.90 & 0.92 & 0.90 & 0.86 & 0.89 & 0.89 \\ \hline
**Manual Augmentation + Augmentation with Borderline-SMOTE** & & & & & & \\ \hline Decision Tree & 92.66 & 0.90 & 0.92 & 0.90 & 0.85 & 0.89 & 0.89 \\ \hline
**Manual Augmentation + Augmentation with Borderline-SMOTE** & & & & & & \\ \hline Decision Tree & 91.33 & 0.89 & 0.92 & 0.88 & 0.82 & 0.87 & 0.86 \\ \hline Random Forest & 90.66 & 0.85 & 0.90 & 0.87 & 0.82 & 0.86 & 0.84 \\ \hline AdaBoosted DT & 92.66 & 0.90 & 0.92 & 0.90 & 0.86 & 0.89 & 0.87 \\ \hline
**Manual Augmentation + Augmentation with Borderline-SMOTE** & & & & & & \\ \hline Decision Tree & 92.66 & 0.90 & 0. As stated above, our comparisons also considered the results without using the framework proposed by [PERSON] et al. [23] (see Tables S2-S4 of our Supplementary Materials). To summarize our findings, we present the decision tree results (from Tables S2-S4 of our Supplementary Materials) in Table 7.  
Besides, we calculated the ROC and PRC curves using a one-against-all classes strategy. We identified the trade-offs between sensitivity (true positive rate) and specificity (true negative rate) to show the model's diagnostic abilities using the ROC area. For example, for the ROC curve performance of the DT model, which relates to the usage of SMOTE and the nested CV methods, high discriminatory power was achieved for all folds. One can also identify that the curves are closer to the upper left corner of each graphic (Figures 3 and 4). In addition, the PRC area shows the relationship between accuracy and recall and is relevant to analyze imbalanced datasets (see Figures S1-S3 of our Supplemental Materials). The precision-recall curve shows the trade-off between precision and recall for different thresholds.