# A New Under-Sampling Method to Face Class Overlap and Imbalance
## 5 Results
### Classification Performance Comparison with State-of-the-Art Methods  
The aim of the first block of this experimental study was to compare the performance of DBMIST-US with that of reference methods to assess its competitiveness (Q1), and also to analyze its robustness across classification models (Q2). Tables A1-A3 in Appendix A report the geometric mean (averaged across the 10 runs) of each under-sampling algorithm using the three classifiers on all the data sets. The classification results on the original (non-preprocessed) data sets were also included as a baseline for performance assessment and usefulness testing. Figure 3 displays the average Friedman rank of the under-sampling algorithm for each classifier. From these graphs, one can observe that DBMIST-US achieved the lowest average ranks regardless of the classifier used, which means that this method performed the best on both the real-life data sets and the synthetic data sets. In addition, as expected, under-sampling the data leads to an improvement in the classification performance, except when using the CNN algorithm; in general, this showed the highest average Friedman ranks (i.e., the worst performance as measured by the geometric mean), especially with the 1 NN and J48 classifiers. It is also interesting to draw attention to the behavior of CNN, NCL, TL, ENN, OSS, RUSBOOST, and SBC with the SVM classification model since the geometric mean obtained by these algorithms was 0 for many of the experimental data sets, as can be seen in Table A3. Taking into account that the geometric mean on the respective non-preprocessed data sets was also 0, this behavior indicates that those under-sampling algorithms could not handle either the class overlap or the class imbalance or even both correctly. To gain some insight into this situation, Figure 4 shows the TPR (red triangles) and TNR (black squares) obtained by using SVM on each under-sampled synthetic data set. From these spider plots, one can observe that the bad performance of most of those methods comes from a true-positive rate equal to 0, which means that the SVM model misclassified all insta  
Figure 3: Comparison of average Friedman ranks. (**a**) real data sets; (**b**) synthetic data sets. Figure 4: True-positive and true-negative rates using SVM on the synthetic data sets. (**a**) RUS; (**b**) CNN; (**c**) NCL; (**d**) TL; (**e**) ENN; (**f**) OSS; (**g**) EUS; (**h**) EE; (**i**) BC; (**j**) RUSBOOST; (**k**) SBC; (**l**) ClusterOSS; (**m**) DBMIST-US. To shed some light onto why the true-positive rate given by those under-sampling algorithms was 0, Figure 5 plots the original paw-50 data set and the under-sampled sets when using either a neighborhood-based method (TL) or a clustering-based method (SBC). One can observe that these algorithms could not solve the class imbalance problem and, consequently, classification by SVM remained biased towards the majority class. On the other hand, the plots show that there still exists some overlapping between classes, indicating that not all noisy negative instances could be removed from the data sets. A particularly interesting problem emerges with the ClusterOSS method where the performance on the minority class improved significantly, but at the cost of causing a severe degradation of the true-negative rate. In Figure 4, the spider plot for the paw-50 data set brings up an especially dramatic case since the TNR was equal to 0. For a deeper understanding of why such a loss of performance in the true-negative rate arose, the scatter plots in Figure 6 display the under-sampled sets when using ClusterOSS and DBMIST-US. As can be viewed, both methods eliminated the problem of class overlap by cleaning the decision boundary completely.