## 4 Experiments
### Implementation and Evaluation
In the hold-out validation method (Table 4), models implemented based on dynamic selection (KNORA-E and KNORA-U) and the DT algorithm presented the highest performances. KNORA-E and KNORA-U achieved the highest accuracy score for the Borderline-SMOTE SVM and Borderline-SMOTE resampling techniques, respectively. The DT model showed stability for all resampling techniques, with an accuracy score of 94.44%. \begin{table}
\begin{tabular}{l c c c c c c} \hline  & **ACC** & **PR** & **Recall** & **Weighted F1** & **Macro F1** & **MCC** & **FM** \\ \hline \hline
**Manual Augmentation Only** & & & & & & & \\ \hline Decision Tree & 94.44 & 0.95 & 0.94 & 0.93 & 0.90 & 0.91 & 0.92 \\ \hline Random Forest & 94.44 & 0.95 & 0.94 & 0.93 & 0.90 & 0.91 & 0.92 \\ AdaBoosted DT & 88.88 & 0.91 & 0.88 & 0.88 & 0.86 & 0.83 & 0.78 \\ OLA & 88.88 & 0.91 & 0.88 & 0.89 & 0.77 & 0.83 & 0.89 \\ LCA & 88.88 & 0.81 & 0.88 & 0.84 & 0.65 & 0.83 & 0.90 \\ KNORA-U & 88.88 & 0.81 & 0.88 & 0.84 & 0.65 & 0.83 & 0.90 \\ KNORA-E & 83.33 & 0.77 & 0.83 & 0.79 & 0.61 & 0.75 & 0.77 \\ \hline
**META-DES** & 83.33 & 0.80 & 0.83 & 0.80 & 0.59 & 0.75 & 0.82 \\ \hline
**Manual Augmentation + Augmentation with SMOTE** & & & & & & & \\ \hline Decision Tree & 94.44 & 0.95 & 0.94 & 0.93 & 0.90 & 0.91 & 0.92 \\ \hline AdaBoosted DT & 94.44 & 0.95 & 0.94 & 0.93 & 0.90 & 0.91 & 0.91 \\ \hline OLA & 88.88 & 0.89 & 0.88 & 0.88 & 0.84 & 0.83 & 0.84 \\ LCA & 88.88 & 0.91 & 0.88 & 0.89 & 0.77 & 0.83 & 0.89 \\ KNORA-U & 94.44 & 0.96 & 0.94 & 0.94 & 0.93 & 0.91 & 0.90 \\ KNORA-E & 94.44 & 0.97 & 0.94 & 0.95 & 0.90 & 0.92 & 0.91 \\ \hline META-DES & 94.44 & 0.96 & 0.94 & 0.94 & 0.93 & 0.91 & 0.90 \\ \hline
**Manual Augmentation + Augmentation with Borderline-SMOTE** & & & & & & \\ \hline Decision Tree & 94.44 & 0.95 & 0.94 & 0.93 & 0.90 & 0.91 & 0.92 \\ \hline Random Forest & 88.88 & 0.91 & 0.88 & 0.88 & 0.86 & 0.83 & 0.78 \\ \hline AdaBoosted DT & 88.88 & 0.89 & 0.88 & 0.88 & 0.86 & 0.82 & 0.80 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Results for the hold-out method for the ensemble models implemented based on the framework proposed by [PERSON] et al. [23]. \begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Importance** & **Low** & **Moderate** & **High** & **Very High** \\ \hline
1 & AH & DM & AH & Crea \\ \hline
2 & DM & Albu & DM & Age \\ \hline
3 & Albu & - & Albu & GFR \\ \hline
4 & Age & - & Age & - \\ \hline
5 & GFR & - & Gender & - \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Results of feature selection for each binary problem generated using the low risk, moderate risk, high risk, and very high risk classes. Finally, we applied the multiple stratified CV method for the ensemble models implemented based on the framework proposed by [PERSON] et al. [23]. \begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & **ACC** & **PR** & **Recall** & **Weighted F1** & **Macro F1** & **MCC** & **FM** \\ \hline OLA & 88.88 & 0.89 & 0.88 & 0.88 & 0.84 & 0.83 & 0.84 \\ \hline LCA & 88.88 & 0.91 & 0.88 & 0.88 & 0.86 & 0.83 & 0.78 \\ \hline KNORA-U & 100.00 & 1.00 & 1.00 & 1.00 & 1.00 & 1.00 & 1.00 \\ \hline KNORA-E & 94.44 & 0.95 & 0.94 & 0.93 & 0.90 & 0.91 & 0.91 \\ \hline META-DES & 88.88 & 0.91 & 0.88 & 0.88 & 0.80 & 0.83 & 0.84 \\ \hline
**Manual Augmentation + Augmentation with Borderline-SMOTE SVM** & & & & & & & \\ \hline Decision Tree & 94.44 & 0.95 & 0.94 & 0.93 & 0.90 & 0.91 & 0.92 \\ \hline Random Forest & 94.44 & 0.96 & 0.94 & 0.94 & 0.93 & 0.91 & 0.90 \\ \hline AdaBoosted DT & 88.88 & 0.91 & 0.88 & 0.88 & 0.86 & 0.83 & 0.78 \\ \hline OLA & 88.88 & 0.89 & 0.88 & 0.88 & 0.84 & 0.83 & 0.84 \\ \hline LCA & 88.88 & 0.92 & 0.88 & 0.88 & 0.79 & 0.83 & 0.84 \\ \hline KNORA-U & 88.88 & 88.88 & 88.88 & 88.88 & 0.84 & 0.82 & 0.84 \\ \hline KNORA-E & 100.00 & 1.00 & 1.00 & 1.00 & 1.00 & 1.00 & 1.00 \\ \hline META-DES & 94.44 & 0.96 & 0.94 & 0.94 & 0.93 & 0.91 & 0.90 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: \(Cont.\)