# A Novel Ensemble Framework Based on K-Means and Resampling for Imbalanced Data
## 4 Experiment Preparation and Result Analysis
### Experimental Results and Analysis  
In order to ensure the fairness of the results, this paper adopts five-fold cross validation to divide the datasets into five parts on average, 80% of which is the training set, the rest is the test set, and the average of ten experimental results is taken as the final result. An important theme to notice, if cross validation is used, the optimal k-valve need be calculated several times and the can be different for different folds. So for each dataset, we calculate the Average Silhouette Coefficient on each fold for 50 times, which is 250 times in total, then recording the times of corresponding \(k\)-value when the Average Silhouette Coefficient reaches the highest, and selecting the best \(k\)-value by the principle that the minority is subordinate to the majority. The process of \(k\)-value determination and data training is independent. Table 3 shows the classification performance of EKR with different base classifiers on 16 datasets. We compared the performance of three base classifiers for EKR, including SVM, Naive Bayes, KNN and C4.5, and recorded Accuracy, F1-score, G-mean, and AUC of EKR with four base classifiers. When the data is extremely imbalanced, Accuracy cannot objectively evaluate the algorithm, it can only be used as a reference index. _F1-score, G-mean_, and AUC are used to evaluate the comprehensive performance of the algorithm, which are more proper evaluation metrics. As the results shown, we can find that C4.5 produces the highest average on all the evaluation metrics, and it maintains the most winner times, followed by SVM, KNN, and Naive Bayes. However, C4.5 does not obtain the best performance on all datasets. On each dataset, we mark the highest value of each evaluation metrics in bold, C4.5 wins the most times. C4.5 can use the unique feature selection method to deal with data samples with more features. SVM can map data to high-dimensional feature space, so it can solve the problem of linear indivisibility in the original space. KNN is the simplest algorithm in machine learning, but it is easy to be affected by surrounding samples when classifying. In this experiment, the kernel function used in SVM is Gaussian radial basis kernel function \(\text{ K}(\text{x},\text{y})=\ e^{\frac{\|\text{x}-y\|^{2}}{2}}\). We set K in KNN to 15. Naive Bayes is not very sensitive to the data with missing features, so the classification effect is poor. As shown in Figures 4-6, we recorded Precision, Recall, and _F1-score_ of EKR with C4.5 for positive samples, negative samples, and overall samples respectively. For positive samples, Recall is higher than Precision, and Recall is more than 89% or even 100% in many datasets, which shows that EKR is suitable for the classification of positive samples. Simultaneously, for negative samples, Precision is higher than Recall, and _F1-score_ of negative samples is better than that of positive samples. Precision and Recall are mutually exclusive, one rise often leads to another fall. For the overall sample of most datasets, there is little difference among Precision, Recall, and _F1-score_. As for datasets D11, D13, D14, D15, and D16, Recall is much higher than Precision obviously. Figures 4-6 prove that the classification ability of EKR for imbalanced datasets. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c c c} \hline \hline \multicolumn{11}{c}{**Dataset**} & \multicolumn{11}{c}{**SYM**} & \multicolumn{11}{c}{**Sams**} & \multicolumn{1}{c}{**Sams**} & \multicolumn{1}{c}{**Sams**} & \multicolumn{1}{c}{**Sams**} & \multicolumn{1}{c}{**Sams**} \\ \hline  
Figure 4: Precision, Recall, and _F1-score_ for the positive samples of 16 datasets. Figure 5: Precision, Recall, and _F1-score_ for the negative samples of 16 datasets. Figure 6: Precision, Recall, and _F1-score_ for the overall samples of 16 datasets.