# Rebalance Weights AdaBoost-SVM Model for Imbalanced Data

Authors: Chunyu Piao; Nan Wang; Chaofeng Yuan
Journal: Computational Intelligence and Neuroscience
 (2023)

---

## Abstract

Classification of imbalanced data is a challenging task that has captured considerable interest in numerous scientific fields by virtue of the great practical value of minority accuracy. Some methods for improving generalization performance have been developed to address this classification situation. Here, we propose a cost-sensitive ensemble learning method using a support vector machine as a base learner of AdaBoost for classifying imbalanced data. Considering that the existing methods are not well studied in terms of how to precisely control the classification accuracy of the minority class, we developed a novel way to rebalance the weights of AdaBoost, and the weights influence the base learner training. This weighting strategy increases the sample weight of the misclassified minority while decreasing the sample weight of the misclassified majority until their distributions are even in each round. Furthermore, we includedP-mean as one of the assessment markers and discussed why it is necessary. Experiments were conducted to compare the proposed and comparison 10 models on 18 datasets in terms of six different metrics. Through comprehensive experimental findings, the statistical study is performed to verify the efficacy and usability of the proposed model.


---

## Body

## 1. Introduction

Classification research is an essential field of study in data science. In balanced data classification, the support vector machine (SVM) [1] and other classification modeling approaches have been broadly discussed and used successfully in a variety of applications [2]. In specific situations where there are imbalanced datasets, the traditional methods always reach their limits in the practical application of classification [3]. They are intended to produce a model that matches the training data well; in an imbalanced dataset, this strategy ignores unusual cases. For this reason, standard classifiers generally perform poorly. With the growth of data mining and data analysis, imbalanced class learning has become a hot topic, and many academics have conducted comprehensive studies on the subject [3, 4]. They defined the nature of imbalance data categorization along three dimensions: concept complexity, training set size, and degree of imbalance between the two classes. They also show how imbalanced datasets can invalidate many conventional classifiers and offer some instances of how to tackle these difficulties. Ghosh et al. attempted to tackle the problem using deep learning systems in 2021, and they discovered that deeper architectures are useful on some data with specific structures in both artificial and real image datasets [5]. Thus far, many excellent research results and improved algorithms have been proposed to solve the problems, such as covering data streams and big data analytics, in this field. Imbalance classification problems include binary imbalance classifications and multiple imbalance classifications that can be transformed into binary problems for solution. Binary imbalance classification problems are not only frequently encountered in real life but are also interesting problems in machine learning (ML). This class of problems is characterized by the fact that the number of samples from one side of the dichotomous dataset involved, called the minority class, is smaller than that of the other side, called the majority class. This minority class is more interested in classification tasks such as medical diagnosis [6]. The identification target detects the people with diseases that belong to the minority, and the consequences of the minority being misclassified are more severe than in the reverse case. The same is valid for detecting images [7], fraud [8], managing risk [9], classifying text [10], and recognizing faces [11]. Binary classification problems with imbalanced data are prevalent compared to other issues in real life. As a basis for classification problems, solutions to binary classification problems can be derived from other classification problems, such as multiple classification problems. Therefore, it is vital to study binary classifications in rare class classifications.In the literature, operating on data or algorithms are the two leading solutions to the problem of imbalanced datasets [12]. The analysis of minority class structure involves determining the imbalance rate and whether it is the overlapping type of distribution; this is the critical reason why the study is complicated. The classification of extremely imbalanced datasets is even more complicated [4]. Resampling techniques for data processing have been adopted to renew the class distribution, such as sampling less of the prevalent class, sampling more of the minority, or more complex techniques [13, 14]. The most popular synthetic minority oversampling technique (SMOTE) is a simple and effective resampling method, which is also widely studied and used in combination with algorithms in the binary imbalance problem. However, resampling may undertake the risk of losing the essential information of the majority, overfitting the minority, and causing the preprocessed dataset to be unlike the raw data. Thus, in many cases, data-level methods are not studied alone but in combination with algorithm-level methods [15–17]. At the algorithmic level, they train the classifier through the data without making distribution changes. Weighting or thresholding support functions or class likelihood estimates can produce better results than resampling the data and can be applied to any conventional classifier [4]. In addition, cost-sensitive learning and ensemble schemes are popular algorithms. A series of cost-sensitive versions of SVM are proposed, for example, cost-sensitive SVM (CS-SVM), which uses SVM to extend its loss function to achieve the objective [18]. SVM based on density weight (DSVM) and improved 2-norm-based density-weighted least squares SVM (IDLSSVM) for binary class imbalanced learning problems [19]. Entropy-based fuzzy twin SVM (EFTWSVM), where fuzzy membership values are assigned based on the entropy values of samples [20]. Entropy-based fuzzy least squares SVM (EFLSSVM) and entropy-based fuzzy least squares twin SVM (EFLSTWSVM) for class imbalanced datasets [21], and other cost-sensitive algorithms [22–26]. However, an appropriate decision boundary cannot be found when the minority samples are sparse [27]. Boosting ensemble approaches overcome learning challenges from imbalanced data classes effectively [28, 29]. AdaBoost [30], the representative boosting strategy, enhances the classification performance of a model by minimizing the error probability. AdaBoost follows the output of the current classifier to modify the sample weight distribution for the next round, and it distinguishes between instances of correct classification when weights are reduced and those of misclassification when weights are increased. However, it does not make further distinctions between different classes of instances, which results in the weights of different classes of instances being increased or decreased in the same way, which is clearly an unsuitable strategy for the imbalanced classification problem. In an imbalanced problem, a good strategy of weighting is one that can distinguish between different instance categories, with which more weight can be given to those relevant instances that have high recognition importance. For this reason, researchers have developed a series of algorithms that try to adjust the weights of instances according to their category labels, which are AdaC [31], CSB1, CSB2 [32], and AdaCost [33].SVM and other algorithms can be embedded into a boosting process and have verified exceptional capabilities [34–36]. Considering the binary classification of the data imbalance, we propose an SVM-based ensemble method that increases the focus on minority accuracy. Our approach is manipulated in two aspects:(1)
Ensure the weights of the misclassified minority and majority samples will not be changed in the same way, because we are concerned with whether the misclassified minority samples can be corrected in the next round. We propose a novel way to rebalance the instances’ weights in each boosting processing, although several methods have been proposed for implementing weight updating in AdaBoost [37]. Thus, the sums of the weights of the misclassified minority and majority samples are balanced with each other.(2)
We use a different approach to vary the parameterC, a crucial parameter for the SVM algorithm, so that each sample receives different costs related to the probability of misclassification, rather than simply dividing the samples into minority and majority categories. The key to this operation is to determine cost items as a function of the weighting of the AdaBoost framework at each iteration, retraining the SVM algorithm for the current iteration by changing the cost for each sample to affect the next iteration positively. Combining these two aspects establishes a link between the AdaBoost frame’s weight and the SVM-based classifier’s cost items. The proposed method uses the AdaBoost weight-adjustment process to solve the data imbalance problem. The rebalanced weights assign the determination of the cost items in SVM learners at each iteration. Different SVM-based learners can be generated to improve the generalization performance based on the previous self-adjusting weights of the instances during the boosting process. Additionally, we have conducted experiments on various UCI ML repository [38] datasets. Apart from some routine evaluation indicators used for binary classification tasks, we use the P-mean to highlight the accuracy of minority classification to display the high efficiency of the presented algorithm.The remainder of this paper is organized as follows. Section2 summarizes the related literature on classifying imbalanced data in recent years. Section 3 introduces the background models, including the SVM model, the enhanced AdaBoost model, and AdaBoost with SVM-based and cost-sensitive SVM. Section 4 presents our method and procedure. Section 5 presents the results and comparisons with other methods based on different datasets and metrics. Section 6 presents the conclusions.


## 2. Related Works

The topic of imbalanced data classification is a difficulty for all researchers, and in addition to a number of successful strategies that have been investigated, researchers are still attempting to address this obstacle using the most recent methodologies. The binary imbalanced job, as the cornerstone of the imbalanced classification problem, arises from numerous real life applications. Hazarika and Gupta presented a novel density-weighted twin SVM (DWTWSVM) for binary imbalance data classification and used density-weighted least squares twin SVM (DWLSTSVM) to boost the computational speed; then, the optimization problem is turned into solving the 2-norm of slack variables and equality constraints [39]. Additionally, using ensemble techniques to handle the imbalanced binary classifications [4]. We intend to address the imbalanced binary classification problem by altering the base classifier of the AdaBoost algorithm, for example, SVM. The SVM algorithm has been considered as one of the most effective classification methods since its introduction, and many excellent research results and improved algorithms have been proposed to solve the classification problem thus far [40]. Because of the vast potential, using SVM as component classifiers is not a new attempt. Many researchers have long focused on combining SVM and ensemble learning methods. Sun et al. [2] analyzed the AdaBoost algorithm and developed three forms of inputting cost terms into the AdaBoost algorithm framework to achieve cost-sensitive purposes. The costs marked the uneven importance of identification between classes and participated in the weight update of AdaBoost. Based on the research of Sun and Kamel, Tao et al. [41] employed cost-sensitive SVMs as base classifiers, while the normal boosting process was modified into a cost-sensitive classifier by presenting a self-adaptive method for determining the cost weight sequences of misclassification. This method allowed for adapting the various contributions of minority samples in the SVM classifier at each round according to the previous classifiers obtained. In this way, different classifiers were generated, thus improving the generalization performance. Lee et al. [28] introduced a weight adjustment factor mechanism of weighted SVM, which was used as a weak learner, and it was proposed for the imbalance data classification target. Instances were classified into four categories: bounded support vector (BSV), support vector (SV), positive noise, and others based on location. They gave different adjustment factors for BSV, SV, and positive noisy instances. In the process of learning a weighted SVM, the weights of instances in the AdaBoost algorithm were multiplied based on the adjustment factors. Wang and Sun [42] proposed an alternative method to improve AdaBoost based on the AD AdaBoost [43] algorithm. The imbalanced ratio of data was a factor and was defined as b = Np/Nn, the majority, and the minority size ratio. In addition to the implementation of the parameters C and the weights, there was also the implementation of the SVM-based ensemble algorithm by changing σ. Li et al. [44] proposed an AdaBoost-SVM method in which the sequence of trained radial basis function SVM (RBFSVM) component classifiers was inserted into the AdaBoost framework. The large σ-values at the beginning were reduced as the boosting iterations proceeded. This allowed a range of RBFSVM component learners with adaptively different parameters, which would have better generalization than the AdaBoost method using SVM component classifiers with fixed σ-values.Additionally, the SVM ensemble models’ high performance has encouraged researchers to develop applications in different fields. For example, Sun et al. [16] proposed a dynamic approach that proved the efficiency of financial distress forecasting with two types of sample imbalances. This method was a forecasting method that combined the time-weighted strategy and AdaBoost with both the SVM-based integration algorithm and an oversampling technique. The results showed that the embedded integration model had significant advantages over the simple base classification model, although both the simple and embedded integration models improved the identification of rare financially distressed samples. In recent years, Liu et al. [45] presented an AdaBoost algorithm that shared SVM with a series of parameter methods to transfer the source task positive and unlabeled learning problem knowledge to the target task. The method combined the weak classifiers into a strong AdaBoost model for prediction. In addition, they considered the similarity of fuzzy examples in terms of minority and majority classes to refine the classifier’s decision boundary. Yao et al. [46] solved the class imbalance problem in forecasting corporate credit risk in the supply chain context using the suggested hybrid model, which combined SVM and AdaBoost ensemble models with an artificial imbalance rate model and distinct feature selection approaches. They claimed that the proposed model mitigated the problem of class imbalance. This not only enhanced the sample distribution variety but also made the AdaBoost integration more stable and generalizable. Wei et al. [47] proposed a fault diagnosis algorithm to address the problem of poor accuracy of actuator failure identification under airplane closed-loop control. The algorithm extracted failure features using the aggregate experience model decomposition method and principal component analysis (PCA). Simultaneously, an adaptive SVM method was embedded in the AdaBoost framework to perform classification operations on them. Additionally, SVM-based ensemble methods have gained tremendous application in various areas in recent years [48–52].


## 3. Background Models

The following is the basic form of the binary classification model. Suppose a binary classification training dataset is given in which each sample consists of an instance and label asS = {(x1, y1), (x2, y2) ,..., (xN, yN)} instance xi ∈ X⊆Rn, label yi ∈ Y = {−1, +1}. X is the instance space, and Y is the label collection. According to the mathematical representation in the algorithms’ derivation and implementation, we used positive and negative classes to refer to minority and majority classes, respectively.


### 3.1. SVM Model

The SVM learning strategy maximizes the interval, which is called the margin. A wider margin corresponds to a more significant difference between the two types, making it easier for us to distinguish between them. Therefore, finding the optimal decision hyperplane corresponds to the maximum margin between the two types of samples. The SVM model designs a hyperplane with dimensionsm − 1. The hyperplane can divide the data of N samples in m dimensions into two categories. For nonlinearly separable data that can cause problems with the algorithm, two approaches can solve this issue. The first one is to enhance the low-dimensional data through a kernel function and use the SVM model in high dimensions to find the appropriate decision hyperplane. The second method introduces slack variables violating the interval constraints slightly. Soft margin SVM can be converted to optimize:(1)minw,ξ12wTw+C∑iξi,s.t.yiwT⋅xi+b≥1−ξi,ξi≥0,i=1,2,…N.Equation (1) is generally transformed into its dual problem and then solved. After the learning problem of SVM is transformed into convex quadratic programming, it has a globally optimal solution. Several optimization algorithms exist for the fast implementation of this problem. Here, we use the sequential minimum optimization (SMO) algorithm. The essential principle behind this approach is that if all variable solutions fulfill the Karush-Kuhn-Tucker (KKT) condition of this optimization issue, the optimization issue’s solution is attained because KKT is both a necessary and sufficient condition for it; otherwise, two variables are selected, additional variables are specified, and a quadratic programming problem is created for these two variables. The subproblem has two variables: one violates the KKT condition negatively, while the other is found automatically by the restrictions. In this approach, the SMO algorithm constantly decomposes and solves the original issue into subproblems. Through these, we can build a decision function using as follows:(2)fx=sign∑iλiyiKxi,xj+b∗.


### 3.2. Cost-Sensitive Support Vector Machine

Cost-sensitive classification learning approaches eschew the common classification strategy of supposing that the cost of all misclassifications is the same and then design classification algorithms that minimize the probability of error. However, in some of the aforementioned cases, this strategy is suboptimal; for example, one type of error is costlier than the others, or examples from different categories occur with different probabilities. Consequently, it is important to develop extensions of cost-sensitive techniques. Veropoulos et al. [53] presented a penalized regularized cost-sensitive SVM model to reduce the negative overwhelming impact. According to the class labels of the training data, the samples are classified into two exact classes: positive S+ = {i|(xi, yi) ∈ S, yi = 1, i = 1,..., N} and negative S− = {i|(xi, yi) ∈ S, yi = −1, i = 1,..., N}. As the set S is divided into indices S+ and S−, this model also introduces C+ and C− penalty factors for positive and negative slack variables. In the optimization process, the positive samples retained higher penalty values than the negative samples. It implements the SVM problem as follows:(3)minw,ξ12wTw+C+∑iyi=+1Nξi+C−∑iyi=−1Nξi,s.t.yiwT⋅xi+b≥1−ξi,ξi≥0,i=1,2,…N.


### 3.3. Enhanced AdaBoost Model

The accuracy-oriented nature of the AdaBoost algorithm prevents it from achieving the desired results if it is applied directly to the classification of imbalanced data. Thus, several researchers have made various improvements to the AdaBoost framework. The enhanced AdaBoost model [42] can be improved by adding the weighted voting parameters α, which are determined by the overall fault rate and the accuracy of the minority primary interest. Km is the sum of all positive samples’ weights, and Pm is the sum of the sample weights labeled positive and predicted to be positive. The ratio of these two is denoted as γm.(4)Km=∑i:yi=1ωmi,Pm=∑i:yi=1,Gmxi=1ωmi,γm=PmKm.After initializing the sample weightD1, we find the base classifier Gm (x) to minimize the error:(5)em=∑i:Gmxi≠yiωmi.We repeat the computation of the weak classifier weightαm:(6)αm=12ln1−emem+kexpβ2γm−1.Then, we renormalizeωm+1,i until m reaches M, the previous setting iterations. The parameters k and β in the above expression are crucial to ensure that the enhanced AdaBoost boosts the classifying efficiency of S+ and maintains a low global error rate. The final output G (x) is a linear combination of a series of weak classifiers.


### 3.4. AdaBoost with SVM-Based

In imbalance classification problems, the generalization performed by the SVM-based AdaBoost method is superior to that of a single SVM [44]. In this section, we describe the SVM-based AdaBoost model. The AdaBoost model is a forward stepwise additive model consisting of a basic classifier whose loss function is exponential L (y, f (x)) = exp[−y f (x)]. fm − 1 (x) is the first m − 1 base classifier. The αm∗ and Gm∗ that minimize (7) are the αm and Gm (x) obtained by the AdaBoost algorithm:(7)αm,Gmx=minα,G∑iexp−yifm−1xi+αGxi.The algorithm learning model is equivalent to the final classifier of AdaBoost when the base classifier isGm (x):(8)Gx=∑mαmGmx.Because multiple parameters are involved in both SVM and AdaBoost algorithms, there are various combinations of SVM with ensemble learning. These include the cost-sensitive AdaBoost algorithm [2, 31] and AdaC2, which have shown good and relatively stable performance [2]. AdaBoost with a heterogeneous SVM could also work well [54]. The following is a brief and concise example of an SVM cost-sensitive ensemble based on adaptive cost weights [41], a method that adaptively considers the various contributions of positives to the SVM classifier during boosting based on the previously obtained classifiers. This study has done more work on positive class samples because of their greater importance, giving a higher cost value to positive instances that are misclassified than to all correctly classified positive instances. This approach has allowed for a great deal of work on instance placement to be completed, assigning larger cost values to borderline instances rather than instances far from the boundary in all cases where it does not matter whether the positives are classified correctly. By incorporating the costs into the updated weights, the weights of positives with higher costs further increase when they are misclassified; otherwise, they further decrease. Initialize D1 = C+/Z0 for all positive instances, and D1 = C−/Z0 for all negative instances, Z0 = p∗C+ + n∗C−, where p and n are the number of the positive and negative classes. C− defaults to one. We calculate the weight updating parameter as follows:(9)αm=12ln∑i:yi=GmxiCm,iωm,i∑i:yi≠GmxiCm,iωm,i.Further, we update and normalize sample weights:(10)ωm+1,i=Cmiωmiexp−αmyiGmxiZm,(11)Cmi=C+⋅1+11+expgm−1xi,ifyi=1,C−,ifyi=−1.InGmxi=signgmxi, where gxi is the value associated with xi calculated by the decision function of SVM and Zm represents normalization value, the final classifier is Gx=sign∑αm∙Gmxi.


## 3.1. SVM Model

The SVM learning strategy maximizes the interval, which is called the margin. A wider margin corresponds to a more significant difference between the two types, making it easier for us to distinguish between them. Therefore, finding the optimal decision hyperplane corresponds to the maximum margin between the two types of samples. The SVM model designs a hyperplane with dimensionsm − 1. The hyperplane can divide the data of N samples in m dimensions into two categories. For nonlinearly separable data that can cause problems with the algorithm, two approaches can solve this issue. The first one is to enhance the low-dimensional data through a kernel function and use the SVM model in high dimensions to find the appropriate decision hyperplane. The second method introduces slack variables violating the interval constraints slightly. Soft margin SVM can be converted to optimize:(1)minw,ξ12wTw+C∑iξi,s.t.yiwT⋅xi+b≥1−ξi,ξi≥0,i=1,2,…N.Equation (1) is generally transformed into its dual problem and then solved. After the learning problem of SVM is transformed into convex quadratic programming, it has a globally optimal solution. Several optimization algorithms exist for the fast implementation of this problem. Here, we use the sequential minimum optimization (SMO) algorithm. The essential principle behind this approach is that if all variable solutions fulfill the Karush-Kuhn-Tucker (KKT) condition of this optimization issue, the optimization issue’s solution is attained because KKT is both a necessary and sufficient condition for it; otherwise, two variables are selected, additional variables are specified, and a quadratic programming problem is created for these two variables. The subproblem has two variables: one violates the KKT condition negatively, while the other is found automatically by the restrictions. In this approach, the SMO algorithm constantly decomposes and solves the original issue into subproblems. Through these, we can build a decision function using as follows:(2)fx=sign∑iλiyiKxi,xj+b∗.


## 3.2. Cost-Sensitive Support Vector Machine

Cost-sensitive classification learning approaches eschew the common classification strategy of supposing that the cost of all misclassifications is the same and then design classification algorithms that minimize the probability of error. However, in some of the aforementioned cases, this strategy is suboptimal; for example, one type of error is costlier than the others, or examples from different categories occur with different probabilities. Consequently, it is important to develop extensions of cost-sensitive techniques. Veropoulos et al. [53] presented a penalized regularized cost-sensitive SVM model to reduce the negative overwhelming impact. According to the class labels of the training data, the samples are classified into two exact classes: positive S+ = {i|(xi, yi) ∈ S, yi = 1, i = 1,..., N} and negative S− = {i|(xi, yi) ∈ S, yi = −1, i = 1,..., N}. As the set S is divided into indices S+ and S−, this model also introduces C+ and C− penalty factors for positive and negative slack variables. In the optimization process, the positive samples retained higher penalty values than the negative samples. It implements the SVM problem as follows:(3)minw,ξ12wTw+C+∑iyi=+1Nξi+C−∑iyi=−1Nξi,s.t.yiwT⋅xi+b≥1−ξi,ξi≥0,i=1,2,…N.


## 3.3. Enhanced AdaBoost Model

The accuracy-oriented nature of the AdaBoost algorithm prevents it from achieving the desired results if it is applied directly to the classification of imbalanced data. Thus, several researchers have made various improvements to the AdaBoost framework. The enhanced AdaBoost model [42] can be improved by adding the weighted voting parameters α, which are determined by the overall fault rate and the accuracy of the minority primary interest. Km is the sum of all positive samples’ weights, and Pm is the sum of the sample weights labeled positive and predicted to be positive. The ratio of these two is denoted as γm.(4)Km=∑i:yi=1ωmi,Pm=∑i:yi=1,Gmxi=1ωmi,γm=PmKm.After initializing the sample weightD1, we find the base classifier Gm (x) to minimize the error:(5)em=∑i:Gmxi≠yiωmi.We repeat the computation of the weak classifier weightαm:(6)αm=12ln1−emem+kexpβ2γm−1.Then, we renormalizeωm+1,i until m reaches M, the previous setting iterations. The parameters k and β in the above expression are crucial to ensure that the enhanced AdaBoost boosts the classifying efficiency of S+ and maintains a low global error rate. The final output G (x) is a linear combination of a series of weak classifiers.


## 3.4. AdaBoost with SVM-Based

In imbalance classification problems, the generalization performed by the SVM-based AdaBoost method is superior to that of a single SVM [44]. In this section, we describe the SVM-based AdaBoost model. The AdaBoost model is a forward stepwise additive model consisting of a basic classifier whose loss function is exponential L (y, f (x)) = exp[−y f (x)]. fm − 1 (x) is the first m − 1 base classifier. The αm∗ and Gm∗ that minimize (7) are the αm and Gm (x) obtained by the AdaBoost algorithm:(7)αm,Gmx=minα,G∑iexp−yifm−1xi+αGxi.The algorithm learning model is equivalent to the final classifier of AdaBoost when the base classifier isGm (x):(8)Gx=∑mαmGmx.Because multiple parameters are involved in both SVM and AdaBoost algorithms, there are various combinations of SVM with ensemble learning. These include the cost-sensitive AdaBoost algorithm [2, 31] and AdaC2, which have shown good and relatively stable performance [2]. AdaBoost with a heterogeneous SVM could also work well [54]. The following is a brief and concise example of an SVM cost-sensitive ensemble based on adaptive cost weights [41], a method that adaptively considers the various contributions of positives to the SVM classifier during boosting based on the previously obtained classifiers. This study has done more work on positive class samples because of their greater importance, giving a higher cost value to positive instances that are misclassified than to all correctly classified positive instances. This approach has allowed for a great deal of work on instance placement to be completed, assigning larger cost values to borderline instances rather than instances far from the boundary in all cases where it does not matter whether the positives are classified correctly. By incorporating the costs into the updated weights, the weights of positives with higher costs further increase when they are misclassified; otherwise, they further decrease. Initialize D1 = C+/Z0 for all positive instances, and D1 = C−/Z0 for all negative instances, Z0 = p∗C+ + n∗C−, where p and n are the number of the positive and negative classes. C− defaults to one. We calculate the weight updating parameter as follows:(9)αm=12ln∑i:yi=GmxiCm,iωm,i∑i:yi≠GmxiCm,iωm,i.Further, we update and normalize sample weights:(10)ωm+1,i=Cmiωmiexp−αmyiGmxiZm,(11)Cmi=C+⋅1+11+expgm−1xi,ifyi=1,C−,ifyi=−1.InGmxi=signgmxi, where gxi is the value associated with xi calculated by the decision function of SVM and Zm represents normalization value, the final classifier is Gx=sign∑αm∙Gmxi.


## 4. Proposed SVM-Based AdaBoost Ensemble

We use SVM as a fundamental weak learner and extend the weight design of the AdaBoost framework to cost-sensitive classification problems. This extension means rebalancing the sum of positive and negative sample weights that are misclassified. A cost-sensitive ensemble classification algorithm is derived in which the weights of misclassified samples from the positive class are added, and the weights of misclassified samples classified correctly from the prevalent class are reduced in our approach. This guarantees that more weights are cumulated in the positive class to influence the training. Moreover, the update of the sample cost vector by the SVM learner is indirectly determined by the associated weight term. With this strategy, the SVM classifier can adaptively consider the different contributions of each instance in each iteration based on the previous boosting process. Instead of focusing too much on samples from the positive class as in other cost-sensitive class algorithms, our algorithm assigns a higher cost value to all misclassified cases during the SVM training process and further handles misclassified samples from the positive class in the rebalancing phase. This allows our algorithm to not only impact significantly on forming the classification but also to address the cost-sensitivity problem. In this section, we detail the two improvements we made to AdaBoost with SVM and the theoretical justification for each part, giving the algorithmic description at the end. Figure1 shows the procedure of the proposed approach. The specific procedure and detailed computational equations for solving our dual problem are shown in Algorithm 1.Figure 1
The procedure of the proposed approach.Algorithm 1: Proposed SVM-based ensemble algorithm.
(1)
Input: Training samplesS = {(x1, y1), (x2, y2) ,..., (xN, yN)}.(2)
Initialize theD1 = (ω11,..., ω1i,..., ω1N) with ω1i = 1/N, and the C1 = (C11,..., C1i,..., C1N).(3)
Form = 1, 2, …, M.(a)
UsingCm to obtain an SVM classifier Gm (x) on the training dataset.(b)
Calculating the coefficientαm according to equation (15).(c)
Updating rebalancedDm +1 using equations (16), (17).(d)
Updating the value ofCm +1 using equation (14).(4)
Building linear combinations of basic classifiers.fx=∑mαmGmx(5)
Output: The final classifier.Gx=signfx=sign∑mαmGmx


### 4.1. Cost Weights AdaBoost-SVM Model

One way to combat the problem of skewed datasets is to work on the penalty factor, which is to give a larger penalty factor to positive classes with small sample sizes, indicating that we value this part of the sample. The penalty factorC is not a variable, and the whole optimization problem is solved with a value of C that you must specify beforehand. After specifying this value, you can obtain a classifier and then evaluate it with the test data. If the result is not satisfactory, change the value of C and repeat the process. This is a parameter search process; however, it is unlike the optimization problem itself. Here, we determine C automatically by the updated weight values for a reason, not by random guessing.Compared to cost-sensitive SVM, we assign a different cost to every misclassified instance instead of one in distinct classes. To solve the problem efficiently and to apply the kernel technique more conveniently, we convert the primary problem into its dual problem and then solve it. The Veropoulos model of the soft margin SVM prototyping [1, 55] is developed as follows:(12)minw,ξ12wTw+CTξ,s.t.yiwT⋅xi+b≥1−ξi,ξi≥0,i=1,2,⋯N.C is a column vector, C = [C1, C2 ... CN]T. ξ denotes a vector of corresponding slack variables of the training data ξ = [ξ1, ξ2 ... ξN]T. We take the Gaussian kernel function, and the dual and kernelized formulation can be derived as follows:(13)minλ12∑i∑jλiλjyiyjKxi,xj−∑iλi,s.t.∑iλiyi=0,0≤λi≤Ci,i=1,2,⋯N.To find the optimal solutionλ∗=λ1∗,λ2∗,…,λN∗, we choose a positive component 0 ≤ λj ≤ Cj of λ∗ and compute b∗ = yj − ∑λi∗yiK (xi, xj) to construct a decision function f (x) = sign (∑λi∗yiK (xi, xj) + b∗).We divide all the classified samples into three types. In the collection of the first sample typeM1= {i|Gm (xi) yi = 1}, samples in this set type are all correctly classified and labeled as iM1. In the collection of the second sample type M2 = { i | Gm (xi) = -1, yi = 1}, the positive samples are classified incorrectly as negative, and we label the weight of this type of sample as iM2. These samples are concerning, as we consider positives classified correctly as a superior task, and we need to rebalance these instances. In the collection of the third sample type M3 = {i|Gm (xi) = 1, yi = −1}, the samples in the third set are classified incorrectly as negative samples into the positive. We consider them less important than the second type of misclassification; therefore, they would not be weighed again. We denote these samples as iM3. M3 and M2 are all misclassified samples; M1, M2, and M3 are combined into a complete set. Instead of using a parameter constant C in the standard SVM to control the maximum hyperplane interval in the objective function while ensuring the minimum deviation of instances, we would give a parameter vector C so that each slack variable ξi has a weight Ci. We use a different C for each outlier, which means we value each sample differently. We assign a smaller C to those samples that are inconsequential compared with those instances that are not to be misclassified. We update the cost terms continuously as the weights change during the boosting process. This cost weights the AdaBoostSVM model we call Ada-SVM. The cost item of the i-th sample at the (m+1)-th round is applied as follows:(14)Cm+1,i=Cmi,ifi∈M1,Cmi⋅1+N⋅ωm+1,i,ifi∈M2∪M3.


### 4.2. Rebalance Weights Model for Imbalanced Data

Unlike other cost-sensitive classification algorithms, we construct a new computational rule to assign weights to the instances based on the weight adjustment of the original AdaBoost framework. Here,Gm (xi) is the category of the i-th sample predicted by the base learner SVM at the m-th iteration. αm is the coefficient of the m-th base classifier, and ωmi is the weight of the i-th sample at the m-th iteration. (15) is used to calculate αm.(15)αm=12ln∑iM1ωmi∑iM2+M3ωmi.At the (m+1)-th iteration, the weight of misclassified positive and negative instances is formulated as follows:(16)ωm+1,iM2=b⋅ωmiM2Zmexp−αmyiGmxim≥2.The weight of other samples is formulated as follows:(17)ωm+1,iM2+M3=ωmiM2+M3Zmexp−αmyiGmxi,where(18)b=∑iM3ωmi∑iM2ωmim≥2,(19)Zm=∑iM2b⋅ωmiexp−αmyiGmxi,+∑iM2+M3ωmiexp−αmyiGmxi,(20)Dm+1=ωm+1,1,⋯,ωm+1,i,⋯,ωm+1,N.Theorem 1.
The training error bound for the final classifier of the rebalancing AdaBoost is as follows:(21)1N∑iIGxi≠yi≤1N∑iexp−yifxi≤∏mZm.The proof is shown in Appendix A.Theorem 2.
The training error bound for the binary classification problem rebalancing AdaBoost:(22)1N∑iIGxi≠yi≤1+b2Mexp−2Mγ2,where γ = 1/2 − em.The proof is shown in Appendix B.


## 4.1. Cost Weights AdaBoost-SVM Model

One way to combat the problem of skewed datasets is to work on the penalty factor, which is to give a larger penalty factor to positive classes with small sample sizes, indicating that we value this part of the sample. The penalty factorC is not a variable, and the whole optimization problem is solved with a value of C that you must specify beforehand. After specifying this value, you can obtain a classifier and then evaluate it with the test data. If the result is not satisfactory, change the value of C and repeat the process. This is a parameter search process; however, it is unlike the optimization problem itself. Here, we determine C automatically by the updated weight values for a reason, not by random guessing.Compared to cost-sensitive SVM, we assign a different cost to every misclassified instance instead of one in distinct classes. To solve the problem efficiently and to apply the kernel technique more conveniently, we convert the primary problem into its dual problem and then solve it. The Veropoulos model of the soft margin SVM prototyping [1, 55] is developed as follows:(12)minw,ξ12wTw+CTξ,s.t.yiwT⋅xi+b≥1−ξi,ξi≥0,i=1,2,⋯N.C is a column vector, C = [C1, C2 ... CN]T. ξ denotes a vector of corresponding slack variables of the training data ξ = [ξ1, ξ2 ... ξN]T. We take the Gaussian kernel function, and the dual and kernelized formulation can be derived as follows:(13)minλ12∑i∑jλiλjyiyjKxi,xj−∑iλi,s.t.∑iλiyi=0,0≤λi≤Ci,i=1,2,⋯N.To find the optimal solutionλ∗=λ1∗,λ2∗,…,λN∗, we choose a positive component 0 ≤ λj ≤ Cj of λ∗ and compute b∗ = yj − ∑λi∗yiK (xi, xj) to construct a decision function f (x) = sign (∑λi∗yiK (xi, xj) + b∗).We divide all the classified samples into three types. In the collection of the first sample typeM1= {i|Gm (xi) yi = 1}, samples in this set type are all correctly classified and labeled as iM1. In the collection of the second sample type M2 = { i | Gm (xi) = -1, yi = 1}, the positive samples are classified incorrectly as negative, and we label the weight of this type of sample as iM2. These samples are concerning, as we consider positives classified correctly as a superior task, and we need to rebalance these instances. In the collection of the third sample type M3 = {i|Gm (xi) = 1, yi = −1}, the samples in the third set are classified incorrectly as negative samples into the positive. We consider them less important than the second type of misclassification; therefore, they would not be weighed again. We denote these samples as iM3. M3 and M2 are all misclassified samples; M1, M2, and M3 are combined into a complete set. Instead of using a parameter constant C in the standard SVM to control the maximum hyperplane interval in the objective function while ensuring the minimum deviation of instances, we would give a parameter vector C so that each slack variable ξi has a weight Ci. We use a different C for each outlier, which means we value each sample differently. We assign a smaller C to those samples that are inconsequential compared with those instances that are not to be misclassified. We update the cost terms continuously as the weights change during the boosting process. This cost weights the AdaBoostSVM model we call Ada-SVM. The cost item of the i-th sample at the (m+1)-th round is applied as follows:(14)Cm+1,i=Cmi,ifi∈M1,Cmi⋅1+N⋅ωm+1,i,ifi∈M2∪M3.


## 4.2. Rebalance Weights Model for Imbalanced Data

Unlike other cost-sensitive classification algorithms, we construct a new computational rule to assign weights to the instances based on the weight adjustment of the original AdaBoost framework. Here,Gm (xi) is the category of the i-th sample predicted by the base learner SVM at the m-th iteration. αm is the coefficient of the m-th base classifier, and ωmi is the weight of the i-th sample at the m-th iteration. (15) is used to calculate αm.(15)αm=12ln∑iM1ωmi∑iM2+M3ωmi.At the (m+1)-th iteration, the weight of misclassified positive and negative instances is formulated as follows:(16)ωm+1,iM2=b⋅ωmiM2Zmexp−αmyiGmxim≥2.The weight of other samples is formulated as follows:(17)ωm+1,iM2+M3=ωmiM2+M3Zmexp−αmyiGmxi,where(18)b=∑iM3ωmi∑iM2ωmim≥2,(19)Zm=∑iM2b⋅ωmiexp−αmyiGmxi,+∑iM2+M3ωmiexp−αmyiGmxi,(20)Dm+1=ωm+1,1,⋯,ωm+1,i,⋯,ωm+1,N.Theorem 1.
The training error bound for the final classifier of the rebalancing AdaBoost is as follows:(21)1N∑iIGxi≠yi≤1N∑iexp−yifxi≤∏mZm.The proof is shown in Appendix A.Theorem 2.
The training error bound for the binary classification problem rebalancing AdaBoost:(22)1N∑iIGxi≠yi≤1+b2Mexp−2Mγ2,where γ = 1/2 − em.The proof is shown in Appendix B.


## 5. Experimental

Our cost-sensitive ensemble learning method was driven by the goal of obtaining all positive samples classified correctly and dealing effectively with imbalanced datasets. We illustrate the effectiveness of our method for the imbalanced data classification problem by using experimental data. The proposed method was compared with ten other approaches in six metric dimensions on our selected dataset with different imbalance ratios. Because our method is based on changing the AdaBoost base classifier to SVM, we compare our method with the original AdaBoost and SVM algorithms without any tricks, respectively, to show that combining the two methods makes our method better than both of them. To compare the performance with other improved algorithm-level classification methods, SVM-based cost-sensitive methods are chosen, and these included the prorated cost method of using the inverse of the size of positives and negatives as the penalty constants for the different classes, CS-SVM, and SMOTE + SVM. We also compare our approach with other state-of-the-art ensemble and data-level resampling strategies, namely, Easy Ensemble [56], SMOTEBoost [57], and SMOTEBagging [57]. Ada-SVM is used to prove that the rebalancing trick is working. The decision tree algorithm was used as a comparison algorithm to illustrate that our approach improved the recall for the positive samples and was not traded off by sacrificing other metrics.


### 5.1. Description of Datasets

Fourteen datasets with different numbers of attributes and sample sizes were selected from the UCI for the test to assess the behavior of the suggested method in handling classification tasks of imbalanced datasets There were some missing attribute values in several datasets, and KNN handled missing attribute value processing. Table1 lists a detailed description of the dataset used. All datasets had two output labels, denoting the positive and negative categories. The attributes indicate the size of features in datasets. Imbalanced Ratio (IR) is the ratio of the number of samples in the negative class to the number of samples in the positive class. Generally, the larger the IR value is, the more harmful it is to the performance of traditional classifiers. We selected some datasets with IR in the range of 1 to 50 for our experiments. We believe that the ratio of the number of samples to the number of features may also be an influential factor in the classification of imbalanced data, and we chose datasets with this ratio in the range of 6 to 547 for our experiments. The training set was normalized before training by rescaling each feature to homoskedasticity for SVM-based algorithms, and the rescaled features that did not distort the original distribution were used on the testing dataset. Preprocessing eliminated the numerical difference between each feature X; further, controlling the size of each column of feature X within a specific range made the model prediction performance more accurate.Table 1
Description of the datasets.
DatasetsAttributesInstancesIRInstances: attributesEcoli_cp7336148WDBC30569219Breast cancer9699278Ionosphere34351210Wine313178314WPBC3319436Ecoli_im7336348Hepatitis1815449SPECT22267412Teaching5151430German credit241000542Ecoli_pp7336548Segmentation1921006111Glass69214624Ecoli_imU7336948Ecoli_om73361648Yeast58148433186Pageblock1054739547


### 5.2. Evaluation Metrics

Specific evaluation metrics to observe the model’s performance in each category were introduced to evaluate the classifier’s classification performance.


#### 5.2.1. Confusion Matrix

The confusion matrix in Table2 shows how the classification model made mistakes when making predictions. All cases were recorded in the following four categories: TP, FN, FP, and TN.Table 2
Confusion matrix.
Predicted positivePredicted negativeTrue positiveTPFNTrue negativeFPTN


#### 5.2.2. Accuracy, Precision, Recall, and Specificity

Classification accuracy was the index for evaluating the classifier’s performance. This indicator was suitable for datasets with balanced categories. For lopsided datasets, the accuracy became unreliable. If positive examples: negative examples = 1 : 99, then the classifier would incorrectly predict all positive examples as negative. An accuracy rate of 0.99 was possible; however, this model could not identify positive examples. Precision, recall, and specificity are metrics generally used for binary classification problems.(23)Accuracy=TP+TNTP+TN+FN+FP,Precision=TPTP+FP,Recall=TPTP+FN,Specificity=TNTN+FP.


#### 5.2.3. Receiver Operating Characteristic (ROC) and Area under the ROC Curve (AUC), G-Mean, F1-Score, andP-Mean

The ROC curve was a recurrent evaluation index for the two-class classification problem; it visually compared the performance of different models on the same dataset. When comparing the performance of the two models, if the ROC of one model completely wrapped the ROC of the other, the former was superior to the latter in terms of classification performance. The use of the area under the ROC curve (AUC) can afford a better model. AUC inherited the insensitivity as a scalar of the ROC curve and was often used in these classifications [58].In the imbalanced classification problem, it was not comprehensive to consider any indicator alone; we needed to combine metrics to measure the model’s efficiency. TheG-mean [7] indicator considered the accuracy of both the positive and negative samples. It was different from the overall accuracy and avoided the dominant influence of negative samples on the classification performance. The F1-score value allowed us to focus on small outliers as the harmonic mean of a set of numbers and was biased toward the smallest element in the list. We paid attention to positive samples and metrics biased toward positive samples, such as the recall of positive samples, when comparing the results and evaluating an algorithm. F1-scores were occasionally unreliable because they were influenced by the minimum value rather than the maximum value. Hence, we used the P-mean, and the geometric mean belonged to recall and precision, which was considered positive and were not so biased toward the positive as the F1-score [7]. The reason for using P-mean was to examine how well the algorithm performed in classifying positive samples; this aspect has not been widely studied. We considered the P-mean an essential index to discriminate whether the classification of positive samples was sufficient. Given the increasing emphasis on recall, P-mean reconciled the values of precision and recall, evaluating the model’s performance more comprehensively.Figure2 shows that in some circumstances, the usage of P-mean is required. The Hepatitis dataset is stratified sampled and divided into two sections, with 80% of the data serving as the training set and 20% serving as the testing set. Multiple trials were run with our model and the SMOTEBoost model, and the results were displayed as box plots, with the performance of the two models under the F1-score and P-mean assessment criteria highlighted in magenta and blue, respectively. Evidently, the median of our proposed method is slightly lower than that of SMOTEBoost under the F1-score evaluation criterion but significantly higher than that of SMOTEBoost under the P-mean evaluation criterion, and in fact, our method outperforms SMOTEBoost in the goal of increasing the accuracy of small class samples. Under all assessment criteria, the data findings provided by our technique are clearly more focused than those obtained by the SMOTEBoost method, reflecting the improved stability of our suggested classification model.(24)G−mean=recall⋅specificity,F1−score=2⋅recall⋅precisionrecall+precision,P−mean=recall⋅precision.Figure 2
Comparison between SMOTEBoost and the proposed method for hepatitis classification results onF1-score and P-mean metrics.


### 5.3. Experimental Settings

All methods were compared with the proposed approach to produce a comprehensive evaluation. The data findings depicted by the icons are all averages of the testing data following a five-fold cross-validation. To avoid the effect caused by the number of two-class instances for training and eliminate randomness before training, we adopted stratified sampling to ensure that the samples after the five-folds split had the same imbalance ratio as the entire sample set, and then four-folds are used as training data with the remaining one-fold as the testing data. Cross-validation may result in a skewed test set in severe circumstances of data imbalance, leading to an inaccurate evaluation [5]. As a result, we also carried out balanced testing, with all assessments based on five-fold stratified cross-validation. We consider the data with an IR higher than or equal to five to be highly imbalanced.For a more comprehensive comparison of our method with other methods, we ran two independent groups of tests [3, 4]: (1) 17 datasets to compare our method with other methods in terms of inquiry structure of classes and sample-to-feature ratio, and (2) we derived 12 new datasets, each with 1000 examples, from a random undersampling in the Pageblock dataset with different IR , aiming to explore the effect of IR on our method compared with others while maintaining the remaining conditions. We examine the same IR levels with the datasets in the first group and higher IRs of 36, 40, and 50 at the same data size.In the proposed SVM-based AdaBoost ensemble, three parameters had been prespecified: the parameterσ for the Gaussian kernel, the initial penalty vector C for each instance, and T for the number of iterations. We employed grid-search to predefine the Gaussian width parameters for the SVM classifier to avoid implications of parameters on the performance. We use one as the σ value for all datasets, and for others, including the Ecoli, Breast Cancer, Ionosphere, Teaching, and Pageblock datasets, we use 10.The optimization problem was solved in such a way thatC was always a fixed value. The cost factor C in cost-sensitive learning was determined for a specific reason, such as using the ratio of the number of samples between categories as the cost, similarly to the prorated cost method. Under some circumstances, the relevance of distinguishing distinct samples was described by cost items, and the cost of a specific sample relied on the properties of the unique situation. For example, regarding detecting fraud, the cost of missing a specific fraud case was determined by the amount of money involved [8]. Herein, we set the initial value to one, and the adjustment scale was updated by itself according to the changes in the iteration process.The experiments of datasets presented in Table1 were conducted to investigate the influence of the iteration parameter T on the performance of the proposed approach. This was a critical factor for improving classification performance. The G-mean, F1-score, P-mean, AUC, accuracy, and recall in the panels in Figure 3 demonstrate that the algorithm was convergent and typically had stability after T reached 25 under settled σ. Therefore, we set the parameter T for all ensemble learning to a constant value of 25.Figure 3
The trend of evaluation metrics of the training dataset withT.To ensure consistency across experiments, all classification algorithms involving the SVM parts are written in uniform handwritten code. Decision tree, easy ensemble, SMOTEBoost, SMOTEBagging, SMOTE + SVM, and AdaBoost, which make direct calls to the packages in Python, are algorithms that use default settings. Because the base classifier SVM has uncertainties in the selection of support vectors, the results would be slightly different for each run of our model. The implementation of the proposed method is publicly available in a GitHub repository (https://github.com/PChunyu/SVM-Adaboost-C). The computer configuration used to run all methods in this paper is an Intel (R) Core (TM) i5-7500 CPU @ 3.40 Hz and 3.40 GHz.


### 5.4. Experimental Results and Statistical Tests

In this section, we first show the effectiveness of our proposed rebalancing strategy and the proposed model. We present the experimental results of two groups in three dimensions: structure of classes, sample-to-feature ratio, and IR. At last, we performed hypothesis tests on all metrics between the proposed method and others to illustrate the validity of our model.To confirm the validity of our proposed rebalancing weights approach, we compared the classification performance of Ada-SVM and proposed, and their differences are before and after rebalancing weights. Figure4 shows the trend for accuracy and recall. Our approach’s performance was superior to that of Ada-SVM in terms of recall. After almost 20 iterations, the proposed method stabilized the recall rate at one, and the accuracy rate was the same or lower than Ada-SVM. Our approach improved the recall at the expense of accuracy in some datasets, an inevitable consequence of enhancing the recall rate. The rebalancing scheme contributed to the proposed method’s better generalization performance than that of Ada-SVM in terms of recall.Figure 4
Trends in accuracy and recall before and after rebalancing.The Yeast 5 dataset with multiple feature dimensions was projected into the two-dimensional space using PCA to observe the decision boundary and demonstrate the effectiveness of the proposed ensemble. Figure5 shows a set of classifications using different methods. Among the classification boundaries produced by the presented techniques, the proposed model classifies the best and has the highest overall accuracy while ensuring a recall value of one. The decision boundary for simple noncost-sensitive classifiers, such as no cost SVM, AdaBoost, and SMOTE + SVM, is a plane with no convergence, and although SMOTE + SVM is used as a classification method for minority class accuracy, the cost-insensitive regular SVM classifier is still used after SMOTE generates new data. These classifiers either perform better in accuracy alone or in recall only while ignoring total accuracy. The majority of the ensemble algorithms have irregularly delineated boundaries, which can more accurately identify regions with minority samples. However, evidently, the more detailed delineation of the minority by SMOTEBoost and SMOTEBagging does not guarantee that they achieve the desired results on the test set data, whereas Easy Ensemble does. Under normal circumstances, without any adjustment, the decision boundary of the cost-sensitive classifier for lopsided data was curved toward positive [55]. CS-SVM and prorated cost panels demonstrated the decision boundary’s warp toward the positive and were affected by imbalanced training data, particularly the positives. It was the reason CS-SVM and prorated cost led to poor generalization performance one testing data. Ada-SVM, No-Cost SVM, decision tree, SMOTEBoost, SMOTEBagging, and AdaBoost, regardless of accuracy, do not achieve a high recall value when compared to other techniques. The rest of the methods could correctly classify positive samples, whereas the accuracy was lower than that of the proposed method. Our method produced decision boundaries that did not favor the positive as in other cost-sensitive classification algorithms; it tended to the negative. This is because our proposed method aggravated the misclassified positive samples and improved the base classifier after readjusting the cost term for all misclassified samples. In general, the panel of the suggested approach showed the best fit among all the techniques. The maximum accuracy was achieved when the positive recall reached one.Figure 5
Demonstration of decision boundaries and test accuracy in all methods.


#### 5.4.1. First Group Datasets

Table3 displays the experimental results of the first group; the bolded black values reflect the best values across all techniques. Except for the WDBC, Breast Cancer, Ionosphere, and Yeast5 datasets, where it does not achieve the greatest recall value, our technique has the highest recall values in all 13 datasets. On these four datasets, the difference between the recall value and the greatest recall value of our technique is not significant: 0.9297 (0.9576 for Easy Ensemble), 0.9713 (0.9834 for CS-SVM), 0.9603 (0.9923 for CS-SVM), and 0.8194 (0.9750 for Prorated Cost), respectively. It is also worth noting that our technique obtains a recall of one for both the Wine3, SPECT, and Ecoli_im datasets, as well as an average value of one after five-fold cross-validation, which indicates that the greatest recall value is obtained for each of the five cross-validation trials, demonstrating the excellent efficiency of our technique. Some algorithms also showed significant accuracy-orientedness in the experiments; the more lopsided the data, the worse the performance of the no cost, decision tree, and AdaBoost algorithms. As one of the most classic and simple methods, Easy Ensemble has the best performance in every metric on the WDBC dataset and the highest performance in the G-mean, F1-score, accuracy, and AUC metrics on the Ecoli_cp and Breast Cancer datasets. These algorithms in combination with SMOTE have excellent figures in other aspects except the recall metric. In some circumstances, CS-SVM obtained very impressive results as one of the representations of cost-sensitive classifiers at the algorithm level. CS-SVM, on the other hand, trained the classifier with fixed positive and negative category costs, causing it to perform better exclusively on a specific dataset, and prorated the cost as well. This restricts them to just marginally improved outcomes on specific datasets.Table 3
G-mean, F1-score, accuracy, AUC, and recall performance comparisons of different algorithms on seventeen UCI datasets.
DatasetsIRMethodsG-meanF1-scoreAccuracyAUCRecallEcoli_cp1Proposed0.81950.82650.81830.86290.9724Ada-SVM0.81620.81070.82150.85610.8956CS-SVM0.62420.62450.70520.95050.8069Prorated cost0.65580.78070.74090.97980.9722No cost0.68330.73810.72030.91970.8828Decision tree0.90710.89520.91370.90890.8744Easy ensemble0.95820.95170.95830.98900.9581SMOTEBoost0.92790.92080.93150.97380.9091SMOTEBagging0.95030.94390.95250.98310.9372SMOTE + SVM0.78540.81500.79520.91870.9073AdaBoost0.95680.95110.95830.97820.9512WDBC2Proposed0.92740.90500.92790.96820.9297Ada-SVM0.91320.89620.92790.94980.8771CS-SVM0.77160.77840.77820.97570.9105Prorated cost0.87150.85230.90690.98760.7969No cost0.67440.64980.74370.89330.6082Decision tree0.91020.88820.91750.91110.8866Easy ensemble0.96610.95760.96840.99010.9576SMOTEBoost0.95130.94060.95610.98850.9339SMOTEBagging0.95510.94220.95610.98840.9527SMOTE + SVM0.75310.73030.78430.93790.6750AdaBoost0.59740.63520.61270.66660.6976Breast cancer2Proposed0.95230.92710.94710.98450.9713Ada-SVM0.94220.91990.94280.98500.9418CS-SVM0.94940.91910.93990.99030.9834Prorated cost0.95200.93260.95280.98990.9502No cost0.63000.57220.81100.76880.4639Decision tree0.91570.89560.92990.91700.8754Easy ensemble0.96990.95370.96710.99370.9793SMOTEBoost0.93740.92070.94570.98860.9132SMOTEBagging0.95330.93800.95700.98940.9419SMOTE + SVM0.71810.75400.74780.82320.6992AdaBoost0.94920.93540.95570.99030.9297Ionosphere2Proposed0.88130.84720.86590.93440.9603Ada-SVM0.78410.74500.88030.97400.7372CS-SVM0.80970.76350.78050.98050.9923Prorated cost0.92160.88560.90890.97970.9760No cost0.59170.51010.62660.70680.5382Decision tree0.88140.85170.89460.88280.8412Easy ensemble0.91070.88690.91740.97040.8889SMOTEBoost0.90860.88700.92020.95150.8729SMOTEBagging0.90240.87630.91180.96860.8735SMOTE + SVM0.43840.49300.60000.76210.5111AdaBoost0.88850.88400.89110.95960.8311Wine33Proposed0.94360.88010.92110.98471.0000Ada-SVM0.94170.87900.92670.97010.9778CS-SVM0.97410.95120.97210.99670.9800Prorated cost0.93180.89520.93870.98850.9200No cost0.74470.62790.80330.96790.7822Decision tree0.91540.88420.93860.91850.8756Easy ensemble0.96890.94210.96631.00000.9778SMOTEBoost0.97050.95770.97750.99190.9578SMOTEBagging0.98080.96920.98320.99920.9778SMOTE + SVM0.82660.86480.84230.96570.9923AdaBoost0.96430.94360.96670.98460.9600WPBC3Proposed0.55730.42800.54010.67370.7178Ada-SVM0.44790.32230.71710.59890.3244CS-SVM0.62350.47750.73680.74390.5356Prorated cost0.60190.46560.67180.68190.5711No cost0.45550.39210.51280.55000.7022Decision tree0.52220.36230.67680.57020.3667Easy ensemble0.58900.41140.60120.70460.5933SMOTEBoost0.59050.41370.68210.67960.4733SMOTEBagging0.60480.49090.80360.71130.4111SMOTE + SVM0.39310.54130.50010.58860.6413AdaBoost0.51300.37380.76740.71070.3111Ecoli_im3Proposed0.87640.73880.84830.94220.9342Ada-SVM0.78500.69030.85690.93610.6892CS-SVM0.55610.44830.69630.72490.5267Prorated cost0.38680.32820.54990.69450.5692No cost0.77510.64310.76510.85550.8292Decision tree0.81250.72740.87800.82130.7158Easy ensemble0.87820.75510.86290.94310.9108SMOTEBoost0.84840.74420.87200.90730.8175SMOTEBagging0.82190.73100.87480.94830.7392SMOTE + SVM0.67110.68280.71030.75740.7458AdaBoost0.84390.76750.89280.93610.7683Hepatitis4Proposed0.77710.58240.73380.87860.8810Ada-SVM0.74440.62550.84430.82720.6238CS-SVM0.73570.62450.85030.84120.6000Prorated cost0.62450.47020.80520.79670.5095No cost0.59900.43450.73380.82220.5524Decision tree0.56600.40740.76000.62890.4048Easy ensemble0.76250.58700.77230.83650.7524SMOTEBoost0.72350.59690.83810.84520.5905SMOTEBagging0.72180.61850.85080.87130.5714SMOTE + SVM0.73110.76280.75840.89140.8217AdaBoost0.70210.57730.83720.76340.5571SPECT4Proposed0.48590.42080.41530.78101.0000Ada-SVM0.45520.35360.76040.75410.4182CS-SVM0.74690.54170.71550.84680.8182Prorated cost0.65810.49170.63650.80440.8000No cost0.52540.36950.49440.67570.7636Decision tree0.59310.41490.75300.65120.4364Easy ensemble0.72000.51890.73050.81040.7091SMOTEBoost0.68290.51780.79800.84860.5818SMOTEBagging0.67900.53460.81650.82120.5455SMOTE + SVM0.60870.73520.66520.85840.9293AdaBoost0.69950.57910.83540.83790.5455Teaching4Proposed0.45800.54050.47630.73920.9400Ada-SVM0.62950.53810.67660.71020.5689CS-SVM0.62060.53030.68860.73430.6044Prorated cost0.59370.49550.64950.78190.6156No cost0.48520.49130.50340.66840.7867Decision tree0.68070.58280.73480.69810.5711Easy ensemble0.65050.55680.64170.68550.6933SMOTEBoost0.59620.48190.64280.74240.5289SMOTEBagging0.65110.54990.68840.71840.5711SMOTE + SVM0.45520.58420.53940.61620.7195AdaBoost0.59740.63520.61270.66660.6976German credit5Proposed0.28420.46360.36600.58000.9133Ada-SVM0.51000.37670.70600.67350.3100CS-SVM0.50440.48360.55600.72650.7433Prorated cost0.62280.51700.66600.72750.6267No cost0.52650.42340.59200.62160.5567Decision tree0.62650.49750.68600.63860.5200Easy ensemble0.71170.59760.70200.76680.7400SMOTEBoost0.66520.54990.73400.77090.5433SMOTEBagging0.63980.52160.72060.76490.4967SMOTE + SVM0.52710.68270.59930.67770.8543AdaBoost0.59740.63520.61270.66660.6976Ecoli_pp5Proposed0.84810.62150.78600.97190.9636Ada-SVM0.89410.80100.93440.95650.8436CS-SVM0.72800.66880.79720.93450.7673Prorated cost0.64710.52030.72270.95870.7091No cost0.56290.48070.62570.62810.8127Decision tree0.80130.69230.90760.82180.6964Easy ensemble0.90200.76610.91370.95900.8873SMOTEBoost0.87340.77460.92580.94930.8109SMOTEBagging0.85080.78450.93750.94590.7545SMOTE + SVM0.74940.81410.77650.93400.9613AdaBoost0.84980.77370.93160.92270.7509Segmentation6Proposed0.81150.51240.71730.92800.9853Ada-SVM0.81980.72280.92520.95250.7113CS-SVM0.80310.57290.76800.97210.9027Prorated cost0.74050.53940.85970.99140.7933No cost0.55630.32620.46420.68360.8900Decision tree0.94900.93150.98100.95000.9067Easy ensemble0.96140.88580.96480.99770.9567SMOTEBoost0.99000.98170.99480.99970.9833SMOTEBagging0.96560.96220.98950.98870.9333SMOTE + SVM0.64170.34570.54410.69660.8507AdaBoost0.95990.92870.97950.99340.9333Glass 66Proposed0.94960.81910.93930.95230.9667Ada-SVM0.85720.76440.93890.94430.7800CS-SVM0.60860.55250.61930.93460.9000Prorated cost0.31000.25080.35450.91770.8333No cost0.80530.67470.91620.87480.7333Decision tree0.88010.82120.95340.88920.8000Easy ensemble0.91050.77160.92980.97530.8933SMOTEBoost0.89500.84700.96270.96040.8267SMOTEBagging0.91170.84330.95780.95020.8600SMOTE + SVM0.85260.85820.85680.92520.8595AdaBoost0.86790.82640.96230.94400.7800Ecoli_imU9Proposed0.78650.39960.67550.83650.9714Ada-SVM0.81170.48490.73450.84770.9429CS-SVM0.73230.54340.90180.83210.6000Prorated cost0.80090.53790.86330.88820.7429No cost0.77980.41840.68730.91230.9429Decision tree0.74080.58340.91660.77670.6000Easy ensemble0.86480.56190.85410.93840.8857SMOTEBoost0.78760.59720.90480.92310.6857SMOTEBagging0.78620.62700.91670.92660.6571SMOTE + SVM0.63250.66660.72890.88830.8000AdaBoost0.66290.47920.89590.88710.4857Ecoli_om16Proposed0.87360.45310.78640.88811.0000Ada-SVM0.79770.30200.69960.81690.9500CS-SVM0.84820.65400.87470.90400.8500Prorated cost0.81500.73290.97320.90530.7000No cost0.78620.32710.68820.92430.9500Decision tree0.82180.69920.96440.84050.7000Easy ensemble0.86520.55430.91960.97860.8500SMOTEBoost0.88350.81110.97920.97800.8000SMOTEBagging0.91640.87430.98510.96870.8500SMOTE + SVM0.64430.67650.73910.88280.8031AdaBoost0.80900.70430.97020.98850.7000Yeast 533Proposed0.86620.48370.94480.90570.8194Ada-SVM0.72730.47100.96360.93410.5444CS-SVM0.72290.22880.72790.97220.8278Prorated cost0.90050.37740.84170.98420.9750No cost0.80410.36880.92180.95930.7333Decision tree0.76520.60940.97840.80030.6111Easy ensemble0.93920.51810.94740.98680.9333SMOTEBoost0.91960.69960.97780.98660.8639SMOTEBagging0.87040.74810.98520.96900.7722SMOTE + SVM0.76380.81420.79200.91710.8764AdaBoost0.76290.62640.97980.97960.6139The bold values represent the best values across all techniques.Although our method achieves the best level among all methods on the WPBC, Hepatitis, and German datasets, the recall levels of these datasets are overall low compared to the other datasets, where the recall values are all close to one. This has a lot to do with the structure of classes; a good classification can be obtained even using canonical classifiers from nonoverlapping distributions [4]. We selected the Ecoli_pp and German Credit datasets with consistent imbalance levels and a close sample number to feature number size ratios, as illustrated in Figure 6. After projecting the training and test set sample points of these two datasets into the two-dimensional plane separately, we can find that the two datasets differ significantly in the distribution of sample points. Both on the testing set and the training set, the distribution of positive and negative class samples on the Ecoli_pp dataset overlaps less, while the opposite is true for the German Credit dataset, which leads to the fact that all methods classify Ecoli_pp much better than German Credit on almost all metrics. Clearly, the lower the sample size to feature ratio, the more features are relative to the sample size, and too many features can easily lead to overfitting; additionally, in some models, when the sample size to feature ratio is too low, the model’s prediction becomes worse. However, this is not completely absolute, because it is difficult to control the other characteristics of the dataset at exactly the same level when discussing the sample number to feature number ratio. Hepatitis, for example, has a high ratio; however, because of structural overlaps, it is more impacted by the category structure, and even with a high sample-to-feature ratio, classification scores are still rather low. By comparing the data results, we found that a sample number to feature number ratio between 10 and 20 is a desirable range; for example, for the Wine3 and SPECT datasets with the highest recall, their sample number to feature number ratios are 14 and 12.Figure 6
Comparison structure of classes between Ecoli_pp and German credit on the training and testing sets.The most important is that our method obtained a considerable advantage in the positive recall at various IR. The performance of the recall results is depicted in Figure7. The value on the bar is the sum of the all-dataset’s recall of the corresponding algorithm. The proposed approach had the maximum value of recall (15.93), followed by Easy Ensemble (14.57).Figure 7
Comparison of different methods’ performance on all datasets in terms of recall.Table4 displays the Shapiro-Wilk test findings because not all data followed the normal distribution at 5%. For statistical analysis, we used the parametric T-test and the nonparametric Wilcoxon test independently to examine if there were any obvious differences between the proposed strategy and the other ways in the experiment.Table 4
P values of the Shapiro-Wilk test for model performance measures in the first group dataset.
ModelG-meanF1-scoreP-meanAUCAccuracyRecallProposed0.0020.0280.0360.0130.0320.001Ada-SVM0.0250.1530.2050.0180.0260.120CS-SVM0.7190.5230.7160.0090.9330.102Prorated cost0.1880.1090.0490.0070.1850.225No cost0.1420.2130.0370.1400.5540.410Decision tree0.0730.0940.0880.0600.0070.093Easy ensemble0.0110.0400.0470.0010.0050.024SMOTEBoost0.0320.1670.1460.0040.0220.060SMOTEBagging0.0390.0470.0310.0010.0130.117SMOTE + SVM0.2750.1170.1930.0330.1340.757AdaBoost0.1590.2890.2900.0020.0010.419The bold values represent data that obey normal distribution.The recall value of the proposed model does not only look higher than other methods by visualization; however, by statistical tests, we found that the recall value of our method is statistically significantly higher among other methods, as compared to their respective counterparts. The results are listed in Table5. Bold indicated significant values at the 0.05 level. Statistical results demonstrated that the proposed SVM-based ensemble learning method performed better than a single no-cost standard SVM in all metrics except the AUC and accuracy. The T-test and the Wilcoxon test results showed that our model was significantly better than other models in recall performances at the 5% level, while it is not statistically significantly different from other models in terms of other metrics. However, it is significantly different from SMOTEBoost and SMOTEBagging only in terms of accuracy because our method discards accuracy to some extent in exchange for higher recall.Table 5
P values for all performance measures in the first group datasets.
Indicators/ModelAda-SVMCS-SVMProrated costNo costDecision treeEasy ensembleSMOTEBoostSMOTEBaggingSMOTE + SVMAdaBoostG-meanT0.8340.4550.3500.0350.8500.1470.2930.2840.0650.905Wilcox0.3610.1170.2780.0070.8770.1340.2780.2930.0080.959F1-scoreT0.9060.6580.5190.0180.4300.2400.1140.0570.3280.198Wilcox0.7960.9310.6670.0290.4180.1340.0880.0340.4590.134P-meanT0.6080.5020.4070.0090.8180.3780.2640.1340.4960.441Wilcox0.6420.7180.3800.0140.7700.2490.2350.0950.5700.310AUCT0.8490.7150.7350.0680.0940.2870.2970.2760.2680.882Wilcox0.9310.6420.4180.0520.0520.0680.1340.1430.1740.438AccuracyT0.0940.6650.7520.2700.0280.0510.0190.0060.2770.051Wilcox0.2700.8230.9590.1480.0600.0560.0220.0050.1250.029RecallT0.0000.0010.0000.0000.0000.0220.0010.0010.0010.000Wilcox0.0000.0030.0020.0000.0000.0110.0000.0010.0000.000The bold values represent the parts of the tests that are significant.


#### 5.4.2. Second Group Dataset

To observe how our method performs under different IRs, Table6 presents the second group of experimental findings. They come from a resampling of the same dataset, with only different IRs under the condition that other features are guaranteed to be the same. Our method maintains the highest level even at the IR of 36 and 50; however, the recall value only stays at approximately 0.9.Table 6
G-mean, F1-score, accuracy, AUC, and recall performance comparison of different algorithms on Pageblock.
DatasetsIRMethodsG-meanF1-scoreAccuracyAUCRecallpa11Proposed0.86840.86440.87000.94390.8320Ada-SVM0.86740.86580.86900.94320.8480CS-SVM0.68230.74290.73800.94070.7720Prorated cost0.67470.66670.72900.87260.7000No cost0.46750.36100.59800.86330.2340Decision tree0.93150.93140.93200.93270.9260Easy ensemble0.94090.94120.94100.97890.9440SMOTEBoost0.94880.94920.94900.98010.9520SMOTEBagging0.94860.94980.94900.98780.9620SMOTE + SVM0.42850.43350.51800.65130.4240AdaBoost0.93990.93990.94000.97490.9380pa22Proposed0.90270.86260.90490.95040.8971Ada-SVM0.84210.79750.86960.93390.7735CS-SVM0.75230.70400.81470.96140.7029Prorated cost0.67150.58650.77450.95770.6618No cost0.59070.53780.62650.69500.6059Decision tree0.93700.91870.94610.93750.9118Easy ensemble0.94610.91510.94020.98190.9647SMOTEBoost0.93650.91410.94220.97670.9206SMOTEBagging0.95150.92970.95200.99400.9500SMOTE + SVM0.64350.64080.68380.80860.5515AdaBoost0.92870.91040.94120.97910.8941pa33Proposed0.85450.78490.89020.94400.7922Ada-SVM0.81310.75230.89020.93830.7095CS-SVM0.69850.58430.68140.96260.8784Prorated cost0.65120.54820.73330.94610.6902No cost0.63930.56480.84020.87870.4196Decision tree0.93570.90800.95390.93660.9020Easy ensemble0.93740.87320.93040.97590.9529SMOTEBoost0.93330.88190.93820.97890.9255SMOTEBagging0.94650.91970.95980.99120.9216SMOTE + SVM0.54640.53650.62750.76050.4392AdaBoost0.90820.86980.93630.98050.8588pa44Proposed0.78690.69270.88730.92390.6776Ada-SVM0.74080.65670.88240.88780.5794CS-SVM0.57810.52890.63630.93100.7998Prorated cost0.67370.50370.78240.95220.6878No cost0.57790.44920.81080.83160.3773Decision tree0.89130.83300.93330.89650.8288Easy ensemble0.94170.83560.92250.97590.9755SMOTEBoost0.89390.81660.92350.96590.8484SMOTEBagging0.92270.86310.94310.98250.8920SMOTE + SVM0.55720.55790.61630.69740.4828AdaBoost0.87880.82160.93040.97070.8037pa55Proposed0.87110.81150.94000.95100.7845Ada-SVM0.84680.77700.92900.93790.7431CS-SVM0.80680.60970.76400.96750.9214Prorated cost0.70450.52170.83300.96640.6720No cost0.59710.43100.82200.90210.4661Decision tree0.89090.84710.95200.89660.8135Easy ensemble0.95090.82670.93100.98580.9822SMOTEBoost0.91860.83570.94200.98050.8859SMOTEBagging0.93980.88590.96100.98760.9100SMOTE + SVM0.57980.49980.67810.86440.3828AdaBoost0.89250.83590.94600.97190.8203pa66Proposed0.88830.74620.91200.94630.8603Ada-SVM0.82440.71510.92000.96130.7197CS-SVM0.72350.52760.80500.94630.7158Prorated cost0.76400.53480.83400.95580.7867No cost0.66830.41500.69800.73370.6571Decision tree0.87760.82070.95100.88370.7897Easy ensemble0.93530.79470.93000.98360.9441SMOTEBoost0.92660.85410.95700.95470.8879SMOTEBagging0.90960.84360.95500.97970.8532SMOTE + SVM0.72700.71810.74040.81130.6511AdaBoost0.87940.83010.95400.96100.7901pa99Proposed0.86780.59900.86700.90810.8800Ada-SVM0.72190.60430.93000.93810.5400CS-SVM0.82920.61840.83700.94410.8500Prorated cost0.87150.60520.88800.95460.8600No cost0.54280.24670.65400.67530.4900Decision tree0.91460.85850.97200.91780.8500Easy ensemble0.94740.77970.94600.99020.9500SMOTEBoost0.93560.84530.96700.97890.9000SMOTEBagging0.92730.85950.97200.97960.8800SMOTE + SVM0.51150.52850.60110.71080.5189AdaBoost0.92250.88190.97700.98620.8600pa1616Proposed0.91400.59160.91500.94920.9152Ada-SVM0.80290.68810.96500.95970.6606CS-SVM0.75270.61980.95900.96870.5924Prorated cost0.72280.50620.93300.96250.5833No cost0.75780.36370.82700.88110.7379Decision tree0.83440.75760.97300.84920.7091Easy ensemble0.94540.68760.94900.99100.9455SMOTEBoost0.90650.74520.96600.93190.8470SMOTEBagging0.90290.81300.97700.99120.8318SMOTE + SVM0.37730.46810.54460.64090.5012AdaBoost0.87120.80340.97800.95390.7758pa3636Proposed0.82870.25370.78500.83070.8933Ada-SVM0.66810.38630.95300.86890.5133CS-SVM0.66540.39260.83500.94090.6333Prorated cost0.61310.31400.93700.93990.4400No cost0.68430.31370.92500.94970.5200Decision tree0.68800.53660.97600.74150.4933Easy ensemble0.88670.36110.91800.97370.8800SMOTEBoost0.90070.75190.98500.93410.8400SMOTEBagging0.75900.58220.97600.94890.6000SMOTE + SVM0.74460.72040.76570.88610.6609AdaBoost0.79190.65370.98300.93540.6533pa4040Proposed0.81300.44210.95400.90450.7000Ada-SVM0.68500.45780.97000.91260.4900CS-SVM0.50730.12740.56300.85210.6400Prorated cost0.82310.44300.95700.95050.7500No cost0.69890.29090.93700.84050.5400Decision tree0.70790.55110.98100.76090.5300Easy ensemble0.86880.30420.90400.96080.8400SMOTEBoost0.75040.41290.95900.88090.6000SMOTEBagging0.62230.44620.97600.97700.4140SMOTE + SVM0.66960.63960.70290.79050.5237AdaBoost0.62860.50910.98200.83680.4200pa5050Proposed0.87440.32860.91300.93880.8500Ada-SVM0.67050.28120.93300.67490.5500CS-SVM0.78960.37930.90700.91260.7333Prorated cost0.49400.28190.79800.94860.4500No cost0.80950.26330.87500.94030.8000Decision tree0.72320.62760.98800.77350.5500Easy ensemble0.82760.23960.90200.95030.8000SMOTEBoost0.69370.47150.97900.94460.5167SMOTEBagging0.76030.65000.98900.95790.6167SMOTE + SVM0.63150.57220.72020.85140.4941AdaBoost0.78100.65950.98800.96670.6333The bold values represent the parts of the tests that are significant.The higher the IR of the dataset, the worse the classification performance tends to be for all methods. Our method does not perform as well on the second group datasets as first, although with the same IR as first. Overall, the classical Easy Ensemble and SMOTEBagging algorithms show a series of better performances on the Pageblock dataset. This also implies that our approach seems to be more effective with small datasets. However, we found that the recall value of the proposed model presented in Table7 is significantly better than accuracy-oriented algorithms such as No-Cost SVM, decision tree, and AdaBoost. It also outperforms the recall values of cost-sensitive classifiers such as Ada-SVM, prorated, and SMOTE + SVM. The proposed model does not have statistically significant differences in recall values with CS-SVM, Easy Ensemble, SMOTEBoost, and SMOTEBagging state-of-the-art algorithms.Table 7
P values for recall performance measures in the high IR of the Pageblock dataset.
Indicators/ModelAda-SVMCS-SVMProrated costNo costDecision treeEasy ensembleSMOTEBoostSMOTEBaggingSMOTE + SVMAdaBoostRecallT0.0000.0550.0160.0010.0230.1100.3940.1700.0000.060Wilcox0.0040.0970.0180.0060.0300.1250.5650.2500.0020.035The bold values represent the parts of the tests that are significant.The proposed, Ada-SVM, CS-SVM, prorated cost, no cost, decision tree, easy ensemble, SMOTEBoost, SMOTEBagging, SMOTE + SVM, and AdaBoost algorithms used to classify the Ecoli series dataset take approximately 100 s, 100 s, 30 s, 30 s, 5 s, 0.03 s, 50 s, 3 s, 3 s, 20 s, and 1 s, respectively. The time required for the Pageblock dataset of size 1000 is 1700 s, 1700 s, 200 s, 200 s, 40 s, 6 s, 160 s, 7 s, 7 s, 120 s, and 3 s.


## 5.1. Description of Datasets

Fourteen datasets with different numbers of attributes and sample sizes were selected from the UCI for the test to assess the behavior of the suggested method in handling classification tasks of imbalanced datasets There were some missing attribute values in several datasets, and KNN handled missing attribute value processing. Table1 lists a detailed description of the dataset used. All datasets had two output labels, denoting the positive and negative categories. The attributes indicate the size of features in datasets. Imbalanced Ratio (IR) is the ratio of the number of samples in the negative class to the number of samples in the positive class. Generally, the larger the IR value is, the more harmful it is to the performance of traditional classifiers. We selected some datasets with IR in the range of 1 to 50 for our experiments. We believe that the ratio of the number of samples to the number of features may also be an influential factor in the classification of imbalanced data, and we chose datasets with this ratio in the range of 6 to 547 for our experiments. The training set was normalized before training by rescaling each feature to homoskedasticity for SVM-based algorithms, and the rescaled features that did not distort the original distribution were used on the testing dataset. Preprocessing eliminated the numerical difference between each feature X; further, controlling the size of each column of feature X within a specific range made the model prediction performance more accurate.Table 1
Description of the datasets.
DatasetsAttributesInstancesIRInstances: attributesEcoli_cp7336148WDBC30569219Breast cancer9699278Ionosphere34351210Wine313178314WPBC3319436Ecoli_im7336348Hepatitis1815449SPECT22267412Teaching5151430German credit241000542Ecoli_pp7336548Segmentation1921006111Glass69214624Ecoli_imU7336948Ecoli_om73361648Yeast58148433186Pageblock1054739547


## 5.2. Evaluation Metrics

Specific evaluation metrics to observe the model’s performance in each category were introduced to evaluate the classifier’s classification performance.


### 5.2.1. Confusion Matrix

The confusion matrix in Table2 shows how the classification model made mistakes when making predictions. All cases were recorded in the following four categories: TP, FN, FP, and TN.Table 2
Confusion matrix.
Predicted positivePredicted negativeTrue positiveTPFNTrue negativeFPTN


### 5.2.2. Accuracy, Precision, Recall, and Specificity

Classification accuracy was the index for evaluating the classifier’s performance. This indicator was suitable for datasets with balanced categories. For lopsided datasets, the accuracy became unreliable. If positive examples: negative examples = 1 : 99, then the classifier would incorrectly predict all positive examples as negative. An accuracy rate of 0.99 was possible; however, this model could not identify positive examples. Precision, recall, and specificity are metrics generally used for binary classification problems.(23)Accuracy=TP+TNTP+TN+FN+FP,Precision=TPTP+FP,Recall=TPTP+FN,Specificity=TNTN+FP.


### 5.2.3. Receiver Operating Characteristic (ROC) and Area under the ROC Curve (AUC), G-Mean, F1-Score, andP-Mean

The ROC curve was a recurrent evaluation index for the two-class classification problem; it visually compared the performance of different models on the same dataset. When comparing the performance of the two models, if the ROC of one model completely wrapped the ROC of the other, the former was superior to the latter in terms of classification performance. The use of the area under the ROC curve (AUC) can afford a better model. AUC inherited the insensitivity as a scalar of the ROC curve and was often used in these classifications [58].In the imbalanced classification problem, it was not comprehensive to consider any indicator alone; we needed to combine metrics to measure the model’s efficiency. TheG-mean [7] indicator considered the accuracy of both the positive and negative samples. It was different from the overall accuracy and avoided the dominant influence of negative samples on the classification performance. The F1-score value allowed us to focus on small outliers as the harmonic mean of a set of numbers and was biased toward the smallest element in the list. We paid attention to positive samples and metrics biased toward positive samples, such as the recall of positive samples, when comparing the results and evaluating an algorithm. F1-scores were occasionally unreliable because they were influenced by the minimum value rather than the maximum value. Hence, we used the P-mean, and the geometric mean belonged to recall and precision, which was considered positive and were not so biased toward the positive as the F1-score [7]. The reason for using P-mean was to examine how well the algorithm performed in classifying positive samples; this aspect has not been widely studied. We considered the P-mean an essential index to discriminate whether the classification of positive samples was sufficient. Given the increasing emphasis on recall, P-mean reconciled the values of precision and recall, evaluating the model’s performance more comprehensively.Figure2 shows that in some circumstances, the usage of P-mean is required. The Hepatitis dataset is stratified sampled and divided into two sections, with 80% of the data serving as the training set and 20% serving as the testing set. Multiple trials were run with our model and the SMOTEBoost model, and the results were displayed as box plots, with the performance of the two models under the F1-score and P-mean assessment criteria highlighted in magenta and blue, respectively. Evidently, the median of our proposed method is slightly lower than that of SMOTEBoost under the F1-score evaluation criterion but significantly higher than that of SMOTEBoost under the P-mean evaluation criterion, and in fact, our method outperforms SMOTEBoost in the goal of increasing the accuracy of small class samples. Under all assessment criteria, the data findings provided by our technique are clearly more focused than those obtained by the SMOTEBoost method, reflecting the improved stability of our suggested classification model.(24)G−mean=recall⋅specificity,F1−score=2⋅recall⋅precisionrecall+precision,P−mean=recall⋅precision.Figure 2
Comparison between SMOTEBoost and the proposed method for hepatitis classification results onF1-score and P-mean metrics.


## 5.2.1. Confusion Matrix

The confusion matrix in Table2 shows how the classification model made mistakes when making predictions. All cases were recorded in the following four categories: TP, FN, FP, and TN.Table 2
Confusion matrix.
Predicted positivePredicted negativeTrue positiveTPFNTrue negativeFPTN


## 5.2.2. Accuracy, Precision, Recall, and Specificity

Classification accuracy was the index for evaluating the classifier’s performance. This indicator was suitable for datasets with balanced categories. For lopsided datasets, the accuracy became unreliable. If positive examples: negative examples = 1 : 99, then the classifier would incorrectly predict all positive examples as negative. An accuracy rate of 0.99 was possible; however, this model could not identify positive examples. Precision, recall, and specificity are metrics generally used for binary classification problems.(23)Accuracy=TP+TNTP+TN+FN+FP,Precision=TPTP+FP,Recall=TPTP+FN,Specificity=TNTN+FP.


## 5.2.3. Receiver Operating Characteristic (ROC) and Area under the ROC Curve (AUC), G-Mean, F1-Score, andP-Mean

The ROC curve was a recurrent evaluation index for the two-class classification problem; it visually compared the performance of different models on the same dataset. When comparing the performance of the two models, if the ROC of one model completely wrapped the ROC of the other, the former was superior to the latter in terms of classification performance. The use of the area under the ROC curve (AUC) can afford a better model. AUC inherited the insensitivity as a scalar of the ROC curve and was often used in these classifications [58].In the imbalanced classification problem, it was not comprehensive to consider any indicator alone; we needed to combine metrics to measure the model’s efficiency. TheG-mean [7] indicator considered the accuracy of both the positive and negative samples. It was different from the overall accuracy and avoided the dominant influence of negative samples on the classification performance. The F1-score value allowed us to focus on small outliers as the harmonic mean of a set of numbers and was biased toward the smallest element in the list. We paid attention to positive samples and metrics biased toward positive samples, such as the recall of positive samples, when comparing the results and evaluating an algorithm. F1-scores were occasionally unreliable because they were influenced by the minimum value rather than the maximum value. Hence, we used the P-mean, and the geometric mean belonged to recall and precision, which was considered positive and were not so biased toward the positive as the F1-score [7]. The reason for using P-mean was to examine how well the algorithm performed in classifying positive samples; this aspect has not been widely studied. We considered the P-mean an essential index to discriminate whether the classification of positive samples was sufficient. Given the increasing emphasis on recall, P-mean reconciled the values of precision and recall, evaluating the model’s performance more comprehensively.Figure2 shows that in some circumstances, the usage of P-mean is required. The Hepatitis dataset is stratified sampled and divided into two sections, with 80% of the data serving as the training set and 20% serving as the testing set. Multiple trials were run with our model and the SMOTEBoost model, and the results were displayed as box plots, with the performance of the two models under the F1-score and P-mean assessment criteria highlighted in magenta and blue, respectively. Evidently, the median of our proposed method is slightly lower than that of SMOTEBoost under the F1-score evaluation criterion but significantly higher than that of SMOTEBoost under the P-mean evaluation criterion, and in fact, our method outperforms SMOTEBoost in the goal of increasing the accuracy of small class samples. Under all assessment criteria, the data findings provided by our technique are clearly more focused than those obtained by the SMOTEBoost method, reflecting the improved stability of our suggested classification model.(24)G−mean=recall⋅specificity,F1−score=2⋅recall⋅precisionrecall+precision,P−mean=recall⋅precision.Figure 2
Comparison between SMOTEBoost and the proposed method for hepatitis classification results onF1-score and P-mean metrics.


## 5.3. Experimental Settings

All methods were compared with the proposed approach to produce a comprehensive evaluation. The data findings depicted by the icons are all averages of the testing data following a five-fold cross-validation. To avoid the effect caused by the number of two-class instances for training and eliminate randomness before training, we adopted stratified sampling to ensure that the samples after the five-folds split had the same imbalance ratio as the entire sample set, and then four-folds are used as training data with the remaining one-fold as the testing data. Cross-validation may result in a skewed test set in severe circumstances of data imbalance, leading to an inaccurate evaluation [5]. As a result, we also carried out balanced testing, with all assessments based on five-fold stratified cross-validation. We consider the data with an IR higher than or equal to five to be highly imbalanced.For a more comprehensive comparison of our method with other methods, we ran two independent groups of tests [3, 4]: (1) 17 datasets to compare our method with other methods in terms of inquiry structure of classes and sample-to-feature ratio, and (2) we derived 12 new datasets, each with 1000 examples, from a random undersampling in the Pageblock dataset with different IR , aiming to explore the effect of IR on our method compared with others while maintaining the remaining conditions. We examine the same IR levels with the datasets in the first group and higher IRs of 36, 40, and 50 at the same data size.In the proposed SVM-based AdaBoost ensemble, three parameters had been prespecified: the parameterσ for the Gaussian kernel, the initial penalty vector C for each instance, and T for the number of iterations. We employed grid-search to predefine the Gaussian width parameters for the SVM classifier to avoid implications of parameters on the performance. We use one as the σ value for all datasets, and for others, including the Ecoli, Breast Cancer, Ionosphere, Teaching, and Pageblock datasets, we use 10.The optimization problem was solved in such a way thatC was always a fixed value. The cost factor C in cost-sensitive learning was determined for a specific reason, such as using the ratio of the number of samples between categories as the cost, similarly to the prorated cost method. Under some circumstances, the relevance of distinguishing distinct samples was described by cost items, and the cost of a specific sample relied on the properties of the unique situation. For example, regarding detecting fraud, the cost of missing a specific fraud case was determined by the amount of money involved [8]. Herein, we set the initial value to one, and the adjustment scale was updated by itself according to the changes in the iteration process.The experiments of datasets presented in Table1 were conducted to investigate the influence of the iteration parameter T on the performance of the proposed approach. This was a critical factor for improving classification performance. The G-mean, F1-score, P-mean, AUC, accuracy, and recall in the panels in Figure 3 demonstrate that the algorithm was convergent and typically had stability after T reached 25 under settled σ. Therefore, we set the parameter T for all ensemble learning to a constant value of 25.Figure 3
The trend of evaluation metrics of the training dataset withT.To ensure consistency across experiments, all classification algorithms involving the SVM parts are written in uniform handwritten code. Decision tree, easy ensemble, SMOTEBoost, SMOTEBagging, SMOTE + SVM, and AdaBoost, which make direct calls to the packages in Python, are algorithms that use default settings. Because the base classifier SVM has uncertainties in the selection of support vectors, the results would be slightly different for each run of our model. The implementation of the proposed method is publicly available in a GitHub repository (https://github.com/PChunyu/SVM-Adaboost-C). The computer configuration used to run all methods in this paper is an Intel (R) Core (TM) i5-7500 CPU @ 3.40 Hz and 3.40 GHz.


## 5.4. Experimental Results and Statistical Tests

In this section, we first show the effectiveness of our proposed rebalancing strategy and the proposed model. We present the experimental results of two groups in three dimensions: structure of classes, sample-to-feature ratio, and IR. At last, we performed hypothesis tests on all metrics between the proposed method and others to illustrate the validity of our model.To confirm the validity of our proposed rebalancing weights approach, we compared the classification performance of Ada-SVM and proposed, and their differences are before and after rebalancing weights. Figure4 shows the trend for accuracy and recall. Our approach’s performance was superior to that of Ada-SVM in terms of recall. After almost 20 iterations, the proposed method stabilized the recall rate at one, and the accuracy rate was the same or lower than Ada-SVM. Our approach improved the recall at the expense of accuracy in some datasets, an inevitable consequence of enhancing the recall rate. The rebalancing scheme contributed to the proposed method’s better generalization performance than that of Ada-SVM in terms of recall.Figure 4
Trends in accuracy and recall before and after rebalancing.The Yeast 5 dataset with multiple feature dimensions was projected into the two-dimensional space using PCA to observe the decision boundary and demonstrate the effectiveness of the proposed ensemble. Figure5 shows a set of classifications using different methods. Among the classification boundaries produced by the presented techniques, the proposed model classifies the best and has the highest overall accuracy while ensuring a recall value of one. The decision boundary for simple noncost-sensitive classifiers, such as no cost SVM, AdaBoost, and SMOTE + SVM, is a plane with no convergence, and although SMOTE + SVM is used as a classification method for minority class accuracy, the cost-insensitive regular SVM classifier is still used after SMOTE generates new data. These classifiers either perform better in accuracy alone or in recall only while ignoring total accuracy. The majority of the ensemble algorithms have irregularly delineated boundaries, which can more accurately identify regions with minority samples. However, evidently, the more detailed delineation of the minority by SMOTEBoost and SMOTEBagging does not guarantee that they achieve the desired results on the test set data, whereas Easy Ensemble does. Under normal circumstances, without any adjustment, the decision boundary of the cost-sensitive classifier for lopsided data was curved toward positive [55]. CS-SVM and prorated cost panels demonstrated the decision boundary’s warp toward the positive and were affected by imbalanced training data, particularly the positives. It was the reason CS-SVM and prorated cost led to poor generalization performance one testing data. Ada-SVM, No-Cost SVM, decision tree, SMOTEBoost, SMOTEBagging, and AdaBoost, regardless of accuracy, do not achieve a high recall value when compared to other techniques. The rest of the methods could correctly classify positive samples, whereas the accuracy was lower than that of the proposed method. Our method produced decision boundaries that did not favor the positive as in other cost-sensitive classification algorithms; it tended to the negative. This is because our proposed method aggravated the misclassified positive samples and improved the base classifier after readjusting the cost term for all misclassified samples. In general, the panel of the suggested approach showed the best fit among all the techniques. The maximum accuracy was achieved when the positive recall reached one.Figure 5
Demonstration of decision boundaries and test accuracy in all methods.


### 5.4.1. First Group Datasets

Table3 displays the experimental results of the first group; the bolded black values reflect the best values across all techniques. Except for the WDBC, Breast Cancer, Ionosphere, and Yeast5 datasets, where it does not achieve the greatest recall value, our technique has the highest recall values in all 13 datasets. On these four datasets, the difference between the recall value and the greatest recall value of our technique is not significant: 0.9297 (0.9576 for Easy Ensemble), 0.9713 (0.9834 for CS-SVM), 0.9603 (0.9923 for CS-SVM), and 0.8194 (0.9750 for Prorated Cost), respectively. It is also worth noting that our technique obtains a recall of one for both the Wine3, SPECT, and Ecoli_im datasets, as well as an average value of one after five-fold cross-validation, which indicates that the greatest recall value is obtained for each of the five cross-validation trials, demonstrating the excellent efficiency of our technique. Some algorithms also showed significant accuracy-orientedness in the experiments; the more lopsided the data, the worse the performance of the no cost, decision tree, and AdaBoost algorithms. As one of the most classic and simple methods, Easy Ensemble has the best performance in every metric on the WDBC dataset and the highest performance in the G-mean, F1-score, accuracy, and AUC metrics on the Ecoli_cp and Breast Cancer datasets. These algorithms in combination with SMOTE have excellent figures in other aspects except the recall metric. In some circumstances, CS-SVM obtained very impressive results as one of the representations of cost-sensitive classifiers at the algorithm level. CS-SVM, on the other hand, trained the classifier with fixed positive and negative category costs, causing it to perform better exclusively on a specific dataset, and prorated the cost as well. This restricts them to just marginally improved outcomes on specific datasets.Table 3
G-mean, F1-score, accuracy, AUC, and recall performance comparisons of different algorithms on seventeen UCI datasets.
DatasetsIRMethodsG-meanF1-scoreAccuracyAUCRecallEcoli_cp1Proposed0.81950.82650.81830.86290.9724Ada-SVM0.81620.81070.82150.85610.8956CS-SVM0.62420.62450.70520.95050.8069Prorated cost0.65580.78070.74090.97980.9722No cost0.68330.73810.72030.91970.8828Decision tree0.90710.89520.91370.90890.8744Easy ensemble0.95820.95170.95830.98900.9581SMOTEBoost0.92790.92080.93150.97380.9091SMOTEBagging0.95030.94390.95250.98310.9372SMOTE + SVM0.78540.81500.79520.91870.9073AdaBoost0.95680.95110.95830.97820.9512WDBC2Proposed0.92740.90500.92790.96820.9297Ada-SVM0.91320.89620.92790.94980.8771CS-SVM0.77160.77840.77820.97570.9105Prorated cost0.87150.85230.90690.98760.7969No cost0.67440.64980.74370.89330.6082Decision tree0.91020.88820.91750.91110.8866Easy ensemble0.96610.95760.96840.99010.9576SMOTEBoost0.95130.94060.95610.98850.9339SMOTEBagging0.95510.94220.95610.98840.9527SMOTE + SVM0.75310.73030.78430.93790.6750AdaBoost0.59740.63520.61270.66660.6976Breast cancer2Proposed0.95230.92710.94710.98450.9713Ada-SVM0.94220.91990.94280.98500.9418CS-SVM0.94940.91910.93990.99030.9834Prorated cost0.95200.93260.95280.98990.9502No cost0.63000.57220.81100.76880.4639Decision tree0.91570.89560.92990.91700.8754Easy ensemble0.96990.95370.96710.99370.9793SMOTEBoost0.93740.92070.94570.98860.9132SMOTEBagging0.95330.93800.95700.98940.9419SMOTE + SVM0.71810.75400.74780.82320.6992AdaBoost0.94920.93540.95570.99030.9297Ionosphere2Proposed0.88130.84720.86590.93440.9603Ada-SVM0.78410.74500.88030.97400.7372CS-SVM0.80970.76350.78050.98050.9923Prorated cost0.92160.88560.90890.97970.9760No cost0.59170.51010.62660.70680.5382Decision tree0.88140.85170.89460.88280.8412Easy ensemble0.91070.88690.91740.97040.8889SMOTEBoost0.90860.88700.92020.95150.8729SMOTEBagging0.90240.87630.91180.96860.8735SMOTE + SVM0.43840.49300.60000.76210.5111AdaBoost0.88850.88400.89110.95960.8311Wine33Proposed0.94360.88010.92110.98471.0000Ada-SVM0.94170.87900.92670.97010.9778CS-SVM0.97410.95120.97210.99670.9800Prorated cost0.93180.89520.93870.98850.9200No cost0.74470.62790.80330.96790.7822Decision tree0.91540.88420.93860.91850.8756Easy ensemble0.96890.94210.96631.00000.9778SMOTEBoost0.97050.95770.97750.99190.9578SMOTEBagging0.98080.96920.98320.99920.9778SMOTE + SVM0.82660.86480.84230.96570.9923AdaBoost0.96430.94360.96670.98460.9600WPBC3Proposed0.55730.42800.54010.67370.7178Ada-SVM0.44790.32230.71710.59890.3244CS-SVM0.62350.47750.73680.74390.5356Prorated cost0.60190.46560.67180.68190.5711No cost0.45550.39210.51280.55000.7022Decision tree0.52220.36230.67680.57020.3667Easy ensemble0.58900.41140.60120.70460.5933SMOTEBoost0.59050.41370.68210.67960.4733SMOTEBagging0.60480.49090.80360.71130.4111SMOTE + SVM0.39310.54130.50010.58860.6413AdaBoost0.51300.37380.76740.71070.3111Ecoli_im3Proposed0.87640.73880.84830.94220.9342Ada-SVM0.78500.69030.85690.93610.6892CS-SVM0.55610.44830.69630.72490.5267Prorated cost0.38680.32820.54990.69450.5692No cost0.77510.64310.76510.85550.8292Decision tree0.81250.72740.87800.82130.7158Easy ensemble0.87820.75510.86290.94310.9108SMOTEBoost0.84840.74420.87200.90730.8175SMOTEBagging0.82190.73100.87480.94830.7392SMOTE + SVM0.67110.68280.71030.75740.7458AdaBoost0.84390.76750.89280.93610.7683Hepatitis4Proposed0.77710.58240.73380.87860.8810Ada-SVM0.74440.62550.84430.82720.6238CS-SVM0.73570.62450.85030.84120.6000Prorated cost0.62450.47020.80520.79670.5095No cost0.59900.43450.73380.82220.5524Decision tree0.56600.40740.76000.62890.4048Easy ensemble0.76250.58700.77230.83650.7524SMOTEBoost0.72350.59690.83810.84520.5905SMOTEBagging0.72180.61850.85080.87130.5714SMOTE + SVM0.73110.76280.75840.89140.8217AdaBoost0.70210.57730.83720.76340.5571SPECT4Proposed0.48590.42080.41530.78101.0000Ada-SVM0.45520.35360.76040.75410.4182CS-SVM0.74690.54170.71550.84680.8182Prorated cost0.65810.49170.63650.80440.8000No cost0.52540.36950.49440.67570.7636Decision tree0.59310.41490.75300.65120.4364Easy ensemble0.72000.51890.73050.81040.7091SMOTEBoost0.68290.51780.79800.84860.5818SMOTEBagging0.67900.53460.81650.82120.5455SMOTE + SVM0.60870.73520.66520.85840.9293AdaBoost0.69950.57910.83540.83790.5455Teaching4Proposed0.45800.54050.47630.73920.9400Ada-SVM0.62950.53810.67660.71020.5689CS-SVM0.62060.53030.68860.73430.6044Prorated cost0.59370.49550.64950.78190.6156No cost0.48520.49130.50340.66840.7867Decision tree0.68070.58280.73480.69810.5711Easy ensemble0.65050.55680.64170.68550.6933SMOTEBoost0.59620.48190.64280.74240.5289SMOTEBagging0.65110.54990.68840.71840.5711SMOTE + SVM0.45520.58420.53940.61620.7195AdaBoost0.59740.63520.61270.66660.6976German credit5Proposed0.28420.46360.36600.58000.9133Ada-SVM0.51000.37670.70600.67350.3100CS-SVM0.50440.48360.55600.72650.7433Prorated cost0.62280.51700.66600.72750.6267No cost0.52650.42340.59200.62160.5567Decision tree0.62650.49750.68600.63860.5200Easy ensemble0.71170.59760.70200.76680.7400SMOTEBoost0.66520.54990.73400.77090.5433SMOTEBagging0.63980.52160.72060.76490.4967SMOTE + SVM0.52710.68270.59930.67770.8543AdaBoost0.59740.63520.61270.66660.6976Ecoli_pp5Proposed0.84810.62150.78600.97190.9636Ada-SVM0.89410.80100.93440.95650.8436CS-SVM0.72800.66880.79720.93450.7673Prorated cost0.64710.52030.72270.95870.7091No cost0.56290.48070.62570.62810.8127Decision tree0.80130.69230.90760.82180.6964Easy ensemble0.90200.76610.91370.95900.8873SMOTEBoost0.87340.77460.92580.94930.8109SMOTEBagging0.85080.78450.93750.94590.7545SMOTE + SVM0.74940.81410.77650.93400.9613AdaBoost0.84980.77370.93160.92270.7509Segmentation6Proposed0.81150.51240.71730.92800.9853Ada-SVM0.81980.72280.92520.95250.7113CS-SVM0.80310.57290.76800.97210.9027Prorated cost0.74050.53940.85970.99140.7933No cost0.55630.32620.46420.68360.8900Decision tree0.94900.93150.98100.95000.9067Easy ensemble0.96140.88580.96480.99770.9567SMOTEBoost0.99000.98170.99480.99970.9833SMOTEBagging0.96560.96220.98950.98870.9333SMOTE + SVM0.64170.34570.54410.69660.8507AdaBoost0.95990.92870.97950.99340.9333Glass 66Proposed0.94960.81910.93930.95230.9667Ada-SVM0.85720.76440.93890.94430.7800CS-SVM0.60860.55250.61930.93460.9000Prorated cost0.31000.25080.35450.91770.8333No cost0.80530.67470.91620.87480.7333Decision tree0.88010.82120.95340.88920.8000Easy ensemble0.91050.77160.92980.97530.8933SMOTEBoost0.89500.84700.96270.96040.8267SMOTEBagging0.91170.84330.95780.95020.8600SMOTE + SVM0.85260.85820.85680.92520.8595AdaBoost0.86790.82640.96230.94400.7800Ecoli_imU9Proposed0.78650.39960.67550.83650.9714Ada-SVM0.81170.48490.73450.84770.9429CS-SVM0.73230.54340.90180.83210.6000Prorated cost0.80090.53790.86330.88820.7429No cost0.77980.41840.68730.91230.9429Decision tree0.74080.58340.91660.77670.6000Easy ensemble0.86480.56190.85410.93840.8857SMOTEBoost0.78760.59720.90480.92310.6857SMOTEBagging0.78620.62700.91670.92660.6571SMOTE + SVM0.63250.66660.72890.88830.8000AdaBoost0.66290.47920.89590.88710.4857Ecoli_om16Proposed0.87360.45310.78640.88811.0000Ada-SVM0.79770.30200.69960.81690.9500CS-SVM0.84820.65400.87470.90400.8500Prorated cost0.81500.73290.97320.90530.7000No cost0.78620.32710.68820.92430.9500Decision tree0.82180.69920.96440.84050.7000Easy ensemble0.86520.55430.91960.97860.8500SMOTEBoost0.88350.81110.97920.97800.8000SMOTEBagging0.91640.87430.98510.96870.8500SMOTE + SVM0.64430.67650.73910.88280.8031AdaBoost0.80900.70430.97020.98850.7000Yeast 533Proposed0.86620.48370.94480.90570.8194Ada-SVM0.72730.47100.96360.93410.5444CS-SVM0.72290.22880.72790.97220.8278Prorated cost0.90050.37740.84170.98420.9750No cost0.80410.36880.92180.95930.7333Decision tree0.76520.60940.97840.80030.6111Easy ensemble0.93920.51810.94740.98680.9333SMOTEBoost0.91960.69960.97780.98660.8639SMOTEBagging0.87040.74810.98520.96900.7722SMOTE + SVM0.76380.81420.79200.91710.8764AdaBoost0.76290.62640.97980.97960.6139The bold values represent the best values across all techniques.Although our method achieves the best level among all methods on the WPBC, Hepatitis, and German datasets, the recall levels of these datasets are overall low compared to the other datasets, where the recall values are all close to one. This has a lot to do with the structure of classes; a good classification can be obtained even using canonical classifiers from nonoverlapping distributions [4]. We selected the Ecoli_pp and German Credit datasets with consistent imbalance levels and a close sample number to feature number size ratios, as illustrated in Figure 6. After projecting the training and test set sample points of these two datasets into the two-dimensional plane separately, we can find that the two datasets differ significantly in the distribution of sample points. Both on the testing set and the training set, the distribution of positive and negative class samples on the Ecoli_pp dataset overlaps less, while the opposite is true for the German Credit dataset, which leads to the fact that all methods classify Ecoli_pp much better than German Credit on almost all metrics. Clearly, the lower the sample size to feature ratio, the more features are relative to the sample size, and too many features can easily lead to overfitting; additionally, in some models, when the sample size to feature ratio is too low, the model’s prediction becomes worse. However, this is not completely absolute, because it is difficult to control the other characteristics of the dataset at exactly the same level when discussing the sample number to feature number ratio. Hepatitis, for example, has a high ratio; however, because of structural overlaps, it is more impacted by the category structure, and even with a high sample-to-feature ratio, classification scores are still rather low. By comparing the data results, we found that a sample number to feature number ratio between 10 and 20 is a desirable range; for example, for the Wine3 and SPECT datasets with the highest recall, their sample number to feature number ratios are 14 and 12.Figure 6
Comparison structure of classes between Ecoli_pp and German credit on the training and testing sets.The most important is that our method obtained a considerable advantage in the positive recall at various IR. The performance of the recall results is depicted in Figure7. The value on the bar is the sum of the all-dataset’s recall of the corresponding algorithm. The proposed approach had the maximum value of recall (15.93), followed by Easy Ensemble (14.57).Figure 7
Comparison of different methods’ performance on all datasets in terms of recall.Table4 displays the Shapiro-Wilk test findings because not all data followed the normal distribution at 5%. For statistical analysis, we used the parametric T-test and the nonparametric Wilcoxon test independently to examine if there were any obvious differences between the proposed strategy and the other ways in the experiment.Table 4
P values of the Shapiro-Wilk test for model performance measures in the first group dataset.
ModelG-meanF1-scoreP-meanAUCAccuracyRecallProposed0.0020.0280.0360.0130.0320.001Ada-SVM0.0250.1530.2050.0180.0260.120CS-SVM0.7190.5230.7160.0090.9330.102Prorated cost0.1880.1090.0490.0070.1850.225No cost0.1420.2130.0370.1400.5540.410Decision tree0.0730.0940.0880.0600.0070.093Easy ensemble0.0110.0400.0470.0010.0050.024SMOTEBoost0.0320.1670.1460.0040.0220.060SMOTEBagging0.0390.0470.0310.0010.0130.117SMOTE + SVM0.2750.1170.1930.0330.1340.757AdaBoost0.1590.2890.2900.0020.0010.419The bold values represent data that obey normal distribution.The recall value of the proposed model does not only look higher than other methods by visualization; however, by statistical tests, we found that the recall value of our method is statistically significantly higher among other methods, as compared to their respective counterparts. The results are listed in Table5. Bold indicated significant values at the 0.05 level. Statistical results demonstrated that the proposed SVM-based ensemble learning method performed better than a single no-cost standard SVM in all metrics except the AUC and accuracy. The T-test and the Wilcoxon test results showed that our model was significantly better than other models in recall performances at the 5% level, while it is not statistically significantly different from other models in terms of other metrics. However, it is significantly different from SMOTEBoost and SMOTEBagging only in terms of accuracy because our method discards accuracy to some extent in exchange for higher recall.Table 5
P values for all performance measures in the first group datasets.
Indicators/ModelAda-SVMCS-SVMProrated costNo costDecision treeEasy ensembleSMOTEBoostSMOTEBaggingSMOTE + SVMAdaBoostG-meanT0.8340.4550.3500.0350.8500.1470.2930.2840.0650.905Wilcox0.3610.1170.2780.0070.8770.1340.2780.2930.0080.959F1-scoreT0.9060.6580.5190.0180.4300.2400.1140.0570.3280.198Wilcox0.7960.9310.6670.0290.4180.1340.0880.0340.4590.134P-meanT0.6080.5020.4070.0090.8180.3780.2640.1340.4960.441Wilcox0.6420.7180.3800.0140.7700.2490.2350.0950.5700.310AUCT0.8490.7150.7350.0680.0940.2870.2970.2760.2680.882Wilcox0.9310.6420.4180.0520.0520.0680.1340.1430.1740.438AccuracyT0.0940.6650.7520.2700.0280.0510.0190.0060.2770.051Wilcox0.2700.8230.9590.1480.0600.0560.0220.0050.1250.029RecallT0.0000.0010.0000.0000.0000.0220.0010.0010.0010.000Wilcox0.0000.0030.0020.0000.0000.0110.0000.0010.0000.000The bold values represent the parts of the tests that are significant.


### 5.4.2. Second Group Dataset

To observe how our method performs under different IRs, Table6 presents the second group of experimental findings. They come from a resampling of the same dataset, with only different IRs under the condition that other features are guaranteed to be the same. Our method maintains the highest level even at the IR of 36 and 50; however, the recall value only stays at approximately 0.9.Table 6
G-mean, F1-score, accuracy, AUC, and recall performance comparison of different algorithms on Pageblock.
DatasetsIRMethodsG-meanF1-scoreAccuracyAUCRecallpa11Proposed0.86840.86440.87000.94390.8320Ada-SVM0.86740.86580.86900.94320.8480CS-SVM0.68230.74290.73800.94070.7720Prorated cost0.67470.66670.72900.87260.7000No cost0.46750.36100.59800.86330.2340Decision tree0.93150.93140.93200.93270.9260Easy ensemble0.94090.94120.94100.97890.9440SMOTEBoost0.94880.94920.94900.98010.9520SMOTEBagging0.94860.94980.94900.98780.9620SMOTE + SVM0.42850.43350.51800.65130.4240AdaBoost0.93990.93990.94000.97490.9380pa22Proposed0.90270.86260.90490.95040.8971Ada-SVM0.84210.79750.86960.93390.7735CS-SVM0.75230.70400.81470.96140.7029Prorated cost0.67150.58650.77450.95770.6618No cost0.59070.53780.62650.69500.6059Decision tree0.93700.91870.94610.93750.9118Easy ensemble0.94610.91510.94020.98190.9647SMOTEBoost0.93650.91410.94220.97670.9206SMOTEBagging0.95150.92970.95200.99400.9500SMOTE + SVM0.64350.64080.68380.80860.5515AdaBoost0.92870.91040.94120.97910.8941pa33Proposed0.85450.78490.89020.94400.7922Ada-SVM0.81310.75230.89020.93830.7095CS-SVM0.69850.58430.68140.96260.8784Prorated cost0.65120.54820.73330.94610.6902No cost0.63930.56480.84020.87870.4196Decision tree0.93570.90800.95390.93660.9020Easy ensemble0.93740.87320.93040.97590.9529SMOTEBoost0.93330.88190.93820.97890.9255SMOTEBagging0.94650.91970.95980.99120.9216SMOTE + SVM0.54640.53650.62750.76050.4392AdaBoost0.90820.86980.93630.98050.8588pa44Proposed0.78690.69270.88730.92390.6776Ada-SVM0.74080.65670.88240.88780.5794CS-SVM0.57810.52890.63630.93100.7998Prorated cost0.67370.50370.78240.95220.6878No cost0.57790.44920.81080.83160.3773Decision tree0.89130.83300.93330.89650.8288Easy ensemble0.94170.83560.92250.97590.9755SMOTEBoost0.89390.81660.92350.96590.8484SMOTEBagging0.92270.86310.94310.98250.8920SMOTE + SVM0.55720.55790.61630.69740.4828AdaBoost0.87880.82160.93040.97070.8037pa55Proposed0.87110.81150.94000.95100.7845Ada-SVM0.84680.77700.92900.93790.7431CS-SVM0.80680.60970.76400.96750.9214Prorated cost0.70450.52170.83300.96640.6720No cost0.59710.43100.82200.90210.4661Decision tree0.89090.84710.95200.89660.8135Easy ensemble0.95090.82670.93100.98580.9822SMOTEBoost0.91860.83570.94200.98050.8859SMOTEBagging0.93980.88590.96100.98760.9100SMOTE + SVM0.57980.49980.67810.86440.3828AdaBoost0.89250.83590.94600.97190.8203pa66Proposed0.88830.74620.91200.94630.8603Ada-SVM0.82440.71510.92000.96130.7197CS-SVM0.72350.52760.80500.94630.7158Prorated cost0.76400.53480.83400.95580.7867No cost0.66830.41500.69800.73370.6571Decision tree0.87760.82070.95100.88370.7897Easy ensemble0.93530.79470.93000.98360.9441SMOTEBoost0.92660.85410.95700.95470.8879SMOTEBagging0.90960.84360.95500.97970.8532SMOTE + SVM0.72700.71810.74040.81130.6511AdaBoost0.87940.83010.95400.96100.7901pa99Proposed0.86780.59900.86700.90810.8800Ada-SVM0.72190.60430.93000.93810.5400CS-SVM0.82920.61840.83700.94410.8500Prorated cost0.87150.60520.88800.95460.8600No cost0.54280.24670.65400.67530.4900Decision tree0.91460.85850.97200.91780.8500Easy ensemble0.94740.77970.94600.99020.9500SMOTEBoost0.93560.84530.96700.97890.9000SMOTEBagging0.92730.85950.97200.97960.8800SMOTE + SVM0.51150.52850.60110.71080.5189AdaBoost0.92250.88190.97700.98620.8600pa1616Proposed0.91400.59160.91500.94920.9152Ada-SVM0.80290.68810.96500.95970.6606CS-SVM0.75270.61980.95900.96870.5924Prorated cost0.72280.50620.93300.96250.5833No cost0.75780.36370.82700.88110.7379Decision tree0.83440.75760.97300.84920.7091Easy ensemble0.94540.68760.94900.99100.9455SMOTEBoost0.90650.74520.96600.93190.8470SMOTEBagging0.90290.81300.97700.99120.8318SMOTE + SVM0.37730.46810.54460.64090.5012AdaBoost0.87120.80340.97800.95390.7758pa3636Proposed0.82870.25370.78500.83070.8933Ada-SVM0.66810.38630.95300.86890.5133CS-SVM0.66540.39260.83500.94090.6333Prorated cost0.61310.31400.93700.93990.4400No cost0.68430.31370.92500.94970.5200Decision tree0.68800.53660.97600.74150.4933Easy ensemble0.88670.36110.91800.97370.8800SMOTEBoost0.90070.75190.98500.93410.8400SMOTEBagging0.75900.58220.97600.94890.6000SMOTE + SVM0.74460.72040.76570.88610.6609AdaBoost0.79190.65370.98300.93540.6533pa4040Proposed0.81300.44210.95400.90450.7000Ada-SVM0.68500.45780.97000.91260.4900CS-SVM0.50730.12740.56300.85210.6400Prorated cost0.82310.44300.95700.95050.7500No cost0.69890.29090.93700.84050.5400Decision tree0.70790.55110.98100.76090.5300Easy ensemble0.86880.30420.90400.96080.8400SMOTEBoost0.75040.41290.95900.88090.6000SMOTEBagging0.62230.44620.97600.97700.4140SMOTE + SVM0.66960.63960.70290.79050.5237AdaBoost0.62860.50910.98200.83680.4200pa5050Proposed0.87440.32860.91300.93880.8500Ada-SVM0.67050.28120.93300.67490.5500CS-SVM0.78960.37930.90700.91260.7333Prorated cost0.49400.28190.79800.94860.4500No cost0.80950.26330.87500.94030.8000Decision tree0.72320.62760.98800.77350.5500Easy ensemble0.82760.23960.90200.95030.8000SMOTEBoost0.69370.47150.97900.94460.5167SMOTEBagging0.76030.65000.98900.95790.6167SMOTE + SVM0.63150.57220.72020.85140.4941AdaBoost0.78100.65950.98800.96670.6333The bold values represent the parts of the tests that are significant.The higher the IR of the dataset, the worse the classification performance tends to be for all methods. Our method does not perform as well on the second group datasets as first, although with the same IR as first. Overall, the classical Easy Ensemble and SMOTEBagging algorithms show a series of better performances on the Pageblock dataset. This also implies that our approach seems to be more effective with small datasets. However, we found that the recall value of the proposed model presented in Table7 is significantly better than accuracy-oriented algorithms such as No-Cost SVM, decision tree, and AdaBoost. It also outperforms the recall values of cost-sensitive classifiers such as Ada-SVM, prorated, and SMOTE + SVM. The proposed model does not have statistically significant differences in recall values with CS-SVM, Easy Ensemble, SMOTEBoost, and SMOTEBagging state-of-the-art algorithms.Table 7
P values for recall performance measures in the high IR of the Pageblock dataset.
Indicators/ModelAda-SVMCS-SVMProrated costNo costDecision treeEasy ensembleSMOTEBoostSMOTEBaggingSMOTE + SVMAdaBoostRecallT0.0000.0550.0160.0010.0230.1100.3940.1700.0000.060Wilcox0.0040.0970.0180.0060.0300.1250.5650.2500.0020.035The bold values represent the parts of the tests that are significant.The proposed, Ada-SVM, CS-SVM, prorated cost, no cost, decision tree, easy ensemble, SMOTEBoost, SMOTEBagging, SMOTE + SVM, and AdaBoost algorithms used to classify the Ecoli series dataset take approximately 100 s, 100 s, 30 s, 30 s, 5 s, 0.03 s, 50 s, 3 s, 3 s, 20 s, and 1 s, respectively. The time required for the Pageblock dataset of size 1000 is 1700 s, 1700 s, 200 s, 200 s, 40 s, 6 s, 160 s, 7 s, 7 s, 120 s, and 3 s.


## 5.4.1. First Group Datasets

Table3 displays the experimental results of the first group; the bolded black values reflect the best values across all techniques. Except for the WDBC, Breast Cancer, Ionosphere, and Yeast5 datasets, where it does not achieve the greatest recall value, our technique has the highest recall values in all 13 datasets. On these four datasets, the difference between the recall value and the greatest recall value of our technique is not significant: 0.9297 (0.9576 for Easy Ensemble), 0.9713 (0.9834 for CS-SVM), 0.9603 (0.9923 for CS-SVM), and 0.8194 (0.9750 for Prorated Cost), respectively. It is also worth noting that our technique obtains a recall of one for both the Wine3, SPECT, and Ecoli_im datasets, as well as an average value of one after five-fold cross-validation, which indicates that the greatest recall value is obtained for each of the five cross-validation trials, demonstrating the excellent efficiency of our technique. Some algorithms also showed significant accuracy-orientedness in the experiments; the more lopsided the data, the worse the performance of the no cost, decision tree, and AdaBoost algorithms. As one of the most classic and simple methods, Easy Ensemble has the best performance in every metric on the WDBC dataset and the highest performance in the G-mean, F1-score, accuracy, and AUC metrics on the Ecoli_cp and Breast Cancer datasets. These algorithms in combination with SMOTE have excellent figures in other aspects except the recall metric. In some circumstances, CS-SVM obtained very impressive results as one of the representations of cost-sensitive classifiers at the algorithm level. CS-SVM, on the other hand, trained the classifier with fixed positive and negative category costs, causing it to perform better exclusively on a specific dataset, and prorated the cost as well. This restricts them to just marginally improved outcomes on specific datasets.Table 3
G-mean, F1-score, accuracy, AUC, and recall performance comparisons of different algorithms on seventeen UCI datasets.
DatasetsIRMethodsG-meanF1-scoreAccuracyAUCRecallEcoli_cp1Proposed0.81950.82650.81830.86290.9724Ada-SVM0.81620.81070.82150.85610.8956CS-SVM0.62420.62450.70520.95050.8069Prorated cost0.65580.78070.74090.97980.9722No cost0.68330.73810.72030.91970.8828Decision tree0.90710.89520.91370.90890.8744Easy ensemble0.95820.95170.95830.98900.9581SMOTEBoost0.92790.92080.93150.97380.9091SMOTEBagging0.95030.94390.95250.98310.9372SMOTE + SVM0.78540.81500.79520.91870.9073AdaBoost0.95680.95110.95830.97820.9512WDBC2Proposed0.92740.90500.92790.96820.9297Ada-SVM0.91320.89620.92790.94980.8771CS-SVM0.77160.77840.77820.97570.9105Prorated cost0.87150.85230.90690.98760.7969No cost0.67440.64980.74370.89330.6082Decision tree0.91020.88820.91750.91110.8866Easy ensemble0.96610.95760.96840.99010.9576SMOTEBoost0.95130.94060.95610.98850.9339SMOTEBagging0.95510.94220.95610.98840.9527SMOTE + SVM0.75310.73030.78430.93790.6750AdaBoost0.59740.63520.61270.66660.6976Breast cancer2Proposed0.95230.92710.94710.98450.9713Ada-SVM0.94220.91990.94280.98500.9418CS-SVM0.94940.91910.93990.99030.9834Prorated cost0.95200.93260.95280.98990.9502No cost0.63000.57220.81100.76880.4639Decision tree0.91570.89560.92990.91700.8754Easy ensemble0.96990.95370.96710.99370.9793SMOTEBoost0.93740.92070.94570.98860.9132SMOTEBagging0.95330.93800.95700.98940.9419SMOTE + SVM0.71810.75400.74780.82320.6992AdaBoost0.94920.93540.95570.99030.9297Ionosphere2Proposed0.88130.84720.86590.93440.9603Ada-SVM0.78410.74500.88030.97400.7372CS-SVM0.80970.76350.78050.98050.9923Prorated cost0.92160.88560.90890.97970.9760No cost0.59170.51010.62660.70680.5382Decision tree0.88140.85170.89460.88280.8412Easy ensemble0.91070.88690.91740.97040.8889SMOTEBoost0.90860.88700.92020.95150.8729SMOTEBagging0.90240.87630.91180.96860.8735SMOTE + SVM0.43840.49300.60000.76210.5111AdaBoost0.88850.88400.89110.95960.8311Wine33Proposed0.94360.88010.92110.98471.0000Ada-SVM0.94170.87900.92670.97010.9778CS-SVM0.97410.95120.97210.99670.9800Prorated cost0.93180.89520.93870.98850.9200No cost0.74470.62790.80330.96790.7822Decision tree0.91540.88420.93860.91850.8756Easy ensemble0.96890.94210.96631.00000.9778SMOTEBoost0.97050.95770.97750.99190.9578SMOTEBagging0.98080.96920.98320.99920.9778SMOTE + SVM0.82660.86480.84230.96570.9923AdaBoost0.96430.94360.96670.98460.9600WPBC3Proposed0.55730.42800.54010.67370.7178Ada-SVM0.44790.32230.71710.59890.3244CS-SVM0.62350.47750.73680.74390.5356Prorated cost0.60190.46560.67180.68190.5711No cost0.45550.39210.51280.55000.7022Decision tree0.52220.36230.67680.57020.3667Easy ensemble0.58900.41140.60120.70460.5933SMOTEBoost0.59050.41370.68210.67960.4733SMOTEBagging0.60480.49090.80360.71130.4111SMOTE + SVM0.39310.54130.50010.58860.6413AdaBoost0.51300.37380.76740.71070.3111Ecoli_im3Proposed0.87640.73880.84830.94220.9342Ada-SVM0.78500.69030.85690.93610.6892CS-SVM0.55610.44830.69630.72490.5267Prorated cost0.38680.32820.54990.69450.5692No cost0.77510.64310.76510.85550.8292Decision tree0.81250.72740.87800.82130.7158Easy ensemble0.87820.75510.86290.94310.9108SMOTEBoost0.84840.74420.87200.90730.8175SMOTEBagging0.82190.73100.87480.94830.7392SMOTE + SVM0.67110.68280.71030.75740.7458AdaBoost0.84390.76750.89280.93610.7683Hepatitis4Proposed0.77710.58240.73380.87860.8810Ada-SVM0.74440.62550.84430.82720.6238CS-SVM0.73570.62450.85030.84120.6000Prorated cost0.62450.47020.80520.79670.5095No cost0.59900.43450.73380.82220.5524Decision tree0.56600.40740.76000.62890.4048Easy ensemble0.76250.58700.77230.83650.7524SMOTEBoost0.72350.59690.83810.84520.5905SMOTEBagging0.72180.61850.85080.87130.5714SMOTE + SVM0.73110.76280.75840.89140.8217AdaBoost0.70210.57730.83720.76340.5571SPECT4Proposed0.48590.42080.41530.78101.0000Ada-SVM0.45520.35360.76040.75410.4182CS-SVM0.74690.54170.71550.84680.8182Prorated cost0.65810.49170.63650.80440.8000No cost0.52540.36950.49440.67570.7636Decision tree0.59310.41490.75300.65120.4364Easy ensemble0.72000.51890.73050.81040.7091SMOTEBoost0.68290.51780.79800.84860.5818SMOTEBagging0.67900.53460.81650.82120.5455SMOTE + SVM0.60870.73520.66520.85840.9293AdaBoost0.69950.57910.83540.83790.5455Teaching4Proposed0.45800.54050.47630.73920.9400Ada-SVM0.62950.53810.67660.71020.5689CS-SVM0.62060.53030.68860.73430.6044Prorated cost0.59370.49550.64950.78190.6156No cost0.48520.49130.50340.66840.7867Decision tree0.68070.58280.73480.69810.5711Easy ensemble0.65050.55680.64170.68550.6933SMOTEBoost0.59620.48190.64280.74240.5289SMOTEBagging0.65110.54990.68840.71840.5711SMOTE + SVM0.45520.58420.53940.61620.7195AdaBoost0.59740.63520.61270.66660.6976German credit5Proposed0.28420.46360.36600.58000.9133Ada-SVM0.51000.37670.70600.67350.3100CS-SVM0.50440.48360.55600.72650.7433Prorated cost0.62280.51700.66600.72750.6267No cost0.52650.42340.59200.62160.5567Decision tree0.62650.49750.68600.63860.5200Easy ensemble0.71170.59760.70200.76680.7400SMOTEBoost0.66520.54990.73400.77090.5433SMOTEBagging0.63980.52160.72060.76490.4967SMOTE + SVM0.52710.68270.59930.67770.8543AdaBoost0.59740.63520.61270.66660.6976Ecoli_pp5Proposed0.84810.62150.78600.97190.9636Ada-SVM0.89410.80100.93440.95650.8436CS-SVM0.72800.66880.79720.93450.7673Prorated cost0.64710.52030.72270.95870.7091No cost0.56290.48070.62570.62810.8127Decision tree0.80130.69230.90760.82180.6964Easy ensemble0.90200.76610.91370.95900.8873SMOTEBoost0.87340.77460.92580.94930.8109SMOTEBagging0.85080.78450.93750.94590.7545SMOTE + SVM0.74940.81410.77650.93400.9613AdaBoost0.84980.77370.93160.92270.7509Segmentation6Proposed0.81150.51240.71730.92800.9853Ada-SVM0.81980.72280.92520.95250.7113CS-SVM0.80310.57290.76800.97210.9027Prorated cost0.74050.53940.85970.99140.7933No cost0.55630.32620.46420.68360.8900Decision tree0.94900.93150.98100.95000.9067Easy ensemble0.96140.88580.96480.99770.9567SMOTEBoost0.99000.98170.99480.99970.9833SMOTEBagging0.96560.96220.98950.98870.9333SMOTE + SVM0.64170.34570.54410.69660.8507AdaBoost0.95990.92870.97950.99340.9333Glass 66Proposed0.94960.81910.93930.95230.9667Ada-SVM0.85720.76440.93890.94430.7800CS-SVM0.60860.55250.61930.93460.9000Prorated cost0.31000.25080.35450.91770.8333No cost0.80530.67470.91620.87480.7333Decision tree0.88010.82120.95340.88920.8000Easy ensemble0.91050.77160.92980.97530.8933SMOTEBoost0.89500.84700.96270.96040.8267SMOTEBagging0.91170.84330.95780.95020.8600SMOTE + SVM0.85260.85820.85680.92520.8595AdaBoost0.86790.82640.96230.94400.7800Ecoli_imU9Proposed0.78650.39960.67550.83650.9714Ada-SVM0.81170.48490.73450.84770.9429CS-SVM0.73230.54340.90180.83210.6000Prorated cost0.80090.53790.86330.88820.7429No cost0.77980.41840.68730.91230.9429Decision tree0.74080.58340.91660.77670.6000Easy ensemble0.86480.56190.85410.93840.8857SMOTEBoost0.78760.59720.90480.92310.6857SMOTEBagging0.78620.62700.91670.92660.6571SMOTE + SVM0.63250.66660.72890.88830.8000AdaBoost0.66290.47920.89590.88710.4857Ecoli_om16Proposed0.87360.45310.78640.88811.0000Ada-SVM0.79770.30200.69960.81690.9500CS-SVM0.84820.65400.87470.90400.8500Prorated cost0.81500.73290.97320.90530.7000No cost0.78620.32710.68820.92430.9500Decision tree0.82180.69920.96440.84050.7000Easy ensemble0.86520.55430.91960.97860.8500SMOTEBoost0.88350.81110.97920.97800.8000SMOTEBagging0.91640.87430.98510.96870.8500SMOTE + SVM0.64430.67650.73910.88280.8031AdaBoost0.80900.70430.97020.98850.7000Yeast 533Proposed0.86620.48370.94480.90570.8194Ada-SVM0.72730.47100.96360.93410.5444CS-SVM0.72290.22880.72790.97220.8278Prorated cost0.90050.37740.84170.98420.9750No cost0.80410.36880.92180.95930.7333Decision tree0.76520.60940.97840.80030.6111Easy ensemble0.93920.51810.94740.98680.9333SMOTEBoost0.91960.69960.97780.98660.8639SMOTEBagging0.87040.74810.98520.96900.7722SMOTE + SVM0.76380.81420.79200.91710.8764AdaBoost0.76290.62640.97980.97960.6139The bold values represent the best values across all techniques.Although our method achieves the best level among all methods on the WPBC, Hepatitis, and German datasets, the recall levels of these datasets are overall low compared to the other datasets, where the recall values are all close to one. This has a lot to do with the structure of classes; a good classification can be obtained even using canonical classifiers from nonoverlapping distributions [4]. We selected the Ecoli_pp and German Credit datasets with consistent imbalance levels and a close sample number to feature number size ratios, as illustrated in Figure 6. After projecting the training and test set sample points of these two datasets into the two-dimensional plane separately, we can find that the two datasets differ significantly in the distribution of sample points. Both on the testing set and the training set, the distribution of positive and negative class samples on the Ecoli_pp dataset overlaps less, while the opposite is true for the German Credit dataset, which leads to the fact that all methods classify Ecoli_pp much better than German Credit on almost all metrics. Clearly, the lower the sample size to feature ratio, the more features are relative to the sample size, and too many features can easily lead to overfitting; additionally, in some models, when the sample size to feature ratio is too low, the model’s prediction becomes worse. However, this is not completely absolute, because it is difficult to control the other characteristics of the dataset at exactly the same level when discussing the sample number to feature number ratio. Hepatitis, for example, has a high ratio; however, because of structural overlaps, it is more impacted by the category structure, and even with a high sample-to-feature ratio, classification scores are still rather low. By comparing the data results, we found that a sample number to feature number ratio between 10 and 20 is a desirable range; for example, for the Wine3 and SPECT datasets with the highest recall, their sample number to feature number ratios are 14 and 12.Figure 6
Comparison structure of classes between Ecoli_pp and German credit on the training and testing sets.The most important is that our method obtained a considerable advantage in the positive recall at various IR. The performance of the recall results is depicted in Figure7. The value on the bar is the sum of the all-dataset’s recall of the corresponding algorithm. The proposed approach had the maximum value of recall (15.93), followed by Easy Ensemble (14.57).Figure 7
Comparison of different methods’ performance on all datasets in terms of recall.Table4 displays the Shapiro-Wilk test findings because not all data followed the normal distribution at 5%. For statistical analysis, we used the parametric T-test and the nonparametric Wilcoxon test independently to examine if there were any obvious differences between the proposed strategy and the other ways in the experiment.Table 4
P values of the Shapiro-Wilk test for model performance measures in the first group dataset.
ModelG-meanF1-scoreP-meanAUCAccuracyRecallProposed0.0020.0280.0360.0130.0320.001Ada-SVM0.0250.1530.2050.0180.0260.120CS-SVM0.7190.5230.7160.0090.9330.102Prorated cost0.1880.1090.0490.0070.1850.225No cost0.1420.2130.0370.1400.5540.410Decision tree0.0730.0940.0880.0600.0070.093Easy ensemble0.0110.0400.0470.0010.0050.024SMOTEBoost0.0320.1670.1460.0040.0220.060SMOTEBagging0.0390.0470.0310.0010.0130.117SMOTE + SVM0.2750.1170.1930.0330.1340.757AdaBoost0.1590.2890.2900.0020.0010.419The bold values represent data that obey normal distribution.The recall value of the proposed model does not only look higher than other methods by visualization; however, by statistical tests, we found that the recall value of our method is statistically significantly higher among other methods, as compared to their respective counterparts. The results are listed in Table5. Bold indicated significant values at the 0.05 level. Statistical results demonstrated that the proposed SVM-based ensemble learning method performed better than a single no-cost standard SVM in all metrics except the AUC and accuracy. The T-test and the Wilcoxon test results showed that our model was significantly better than other models in recall performances at the 5% level, while it is not statistically significantly different from other models in terms of other metrics. However, it is significantly different from SMOTEBoost and SMOTEBagging only in terms of accuracy because our method discards accuracy to some extent in exchange for higher recall.Table 5
P values for all performance measures in the first group datasets.
Indicators/ModelAda-SVMCS-SVMProrated costNo costDecision treeEasy ensembleSMOTEBoostSMOTEBaggingSMOTE + SVMAdaBoostG-meanT0.8340.4550.3500.0350.8500.1470.2930.2840.0650.905Wilcox0.3610.1170.2780.0070.8770.1340.2780.2930.0080.959F1-scoreT0.9060.6580.5190.0180.4300.2400.1140.0570.3280.198Wilcox0.7960.9310.6670.0290.4180.1340.0880.0340.4590.134P-meanT0.6080.5020.4070.0090.8180.3780.2640.1340.4960.441Wilcox0.6420.7180.3800.0140.7700.2490.2350.0950.5700.310AUCT0.8490.7150.7350.0680.0940.2870.2970.2760.2680.882Wilcox0.9310.6420.4180.0520.0520.0680.1340.1430.1740.438AccuracyT0.0940.6650.7520.2700.0280.0510.0190.0060.2770.051Wilcox0.2700.8230.9590.1480.0600.0560.0220.0050.1250.029RecallT0.0000.0010.0000.0000.0000.0220.0010.0010.0010.000Wilcox0.0000.0030.0020.0000.0000.0110.0000.0010.0000.000The bold values represent the parts of the tests that are significant.


## 5.4.2. Second Group Dataset

To observe how our method performs under different IRs, Table6 presents the second group of experimental findings. They come from a resampling of the same dataset, with only different IRs under the condition that other features are guaranteed to be the same. Our method maintains the highest level even at the IR of 36 and 50; however, the recall value only stays at approximately 0.9.Table 6
G-mean, F1-score, accuracy, AUC, and recall performance comparison of different algorithms on Pageblock.
DatasetsIRMethodsG-meanF1-scoreAccuracyAUCRecallpa11Proposed0.86840.86440.87000.94390.8320Ada-SVM0.86740.86580.86900.94320.8480CS-SVM0.68230.74290.73800.94070.7720Prorated cost0.67470.66670.72900.87260.7000No cost0.46750.36100.59800.86330.2340Decision tree0.93150.93140.93200.93270.9260Easy ensemble0.94090.94120.94100.97890.9440SMOTEBoost0.94880.94920.94900.98010.9520SMOTEBagging0.94860.94980.94900.98780.9620SMOTE + SVM0.42850.43350.51800.65130.4240AdaBoost0.93990.93990.94000.97490.9380pa22Proposed0.90270.86260.90490.95040.8971Ada-SVM0.84210.79750.86960.93390.7735CS-SVM0.75230.70400.81470.96140.7029Prorated cost0.67150.58650.77450.95770.6618No cost0.59070.53780.62650.69500.6059Decision tree0.93700.91870.94610.93750.9118Easy ensemble0.94610.91510.94020.98190.9647SMOTEBoost0.93650.91410.94220.97670.9206SMOTEBagging0.95150.92970.95200.99400.9500SMOTE + SVM0.64350.64080.68380.80860.5515AdaBoost0.92870.91040.94120.97910.8941pa33Proposed0.85450.78490.89020.94400.7922Ada-SVM0.81310.75230.89020.93830.7095CS-SVM0.69850.58430.68140.96260.8784Prorated cost0.65120.54820.73330.94610.6902No cost0.63930.56480.84020.87870.4196Decision tree0.93570.90800.95390.93660.9020Easy ensemble0.93740.87320.93040.97590.9529SMOTEBoost0.93330.88190.93820.97890.9255SMOTEBagging0.94650.91970.95980.99120.9216SMOTE + SVM0.54640.53650.62750.76050.4392AdaBoost0.90820.86980.93630.98050.8588pa44Proposed0.78690.69270.88730.92390.6776Ada-SVM0.74080.65670.88240.88780.5794CS-SVM0.57810.52890.63630.93100.7998Prorated cost0.67370.50370.78240.95220.6878No cost0.57790.44920.81080.83160.3773Decision tree0.89130.83300.93330.89650.8288Easy ensemble0.94170.83560.92250.97590.9755SMOTEBoost0.89390.81660.92350.96590.8484SMOTEBagging0.92270.86310.94310.98250.8920SMOTE + SVM0.55720.55790.61630.69740.4828AdaBoost0.87880.82160.93040.97070.8037pa55Proposed0.87110.81150.94000.95100.7845Ada-SVM0.84680.77700.92900.93790.7431CS-SVM0.80680.60970.76400.96750.9214Prorated cost0.70450.52170.83300.96640.6720No cost0.59710.43100.82200.90210.4661Decision tree0.89090.84710.95200.89660.8135Easy ensemble0.95090.82670.93100.98580.9822SMOTEBoost0.91860.83570.94200.98050.8859SMOTEBagging0.93980.88590.96100.98760.9100SMOTE + SVM0.57980.49980.67810.86440.3828AdaBoost0.89250.83590.94600.97190.8203pa66Proposed0.88830.74620.91200.94630.8603Ada-SVM0.82440.71510.92000.96130.7197CS-SVM0.72350.52760.80500.94630.7158Prorated cost0.76400.53480.83400.95580.7867No cost0.66830.41500.69800.73370.6571Decision tree0.87760.82070.95100.88370.7897Easy ensemble0.93530.79470.93000.98360.9441SMOTEBoost0.92660.85410.95700.95470.8879SMOTEBagging0.90960.84360.95500.97970.8532SMOTE + SVM0.72700.71810.74040.81130.6511AdaBoost0.87940.83010.95400.96100.7901pa99Proposed0.86780.59900.86700.90810.8800Ada-SVM0.72190.60430.93000.93810.5400CS-SVM0.82920.61840.83700.94410.8500Prorated cost0.87150.60520.88800.95460.8600No cost0.54280.24670.65400.67530.4900Decision tree0.91460.85850.97200.91780.8500Easy ensemble0.94740.77970.94600.99020.9500SMOTEBoost0.93560.84530.96700.97890.9000SMOTEBagging0.92730.85950.97200.97960.8800SMOTE + SVM0.51150.52850.60110.71080.5189AdaBoost0.92250.88190.97700.98620.8600pa1616Proposed0.91400.59160.91500.94920.9152Ada-SVM0.80290.68810.96500.95970.6606CS-SVM0.75270.61980.95900.96870.5924Prorated cost0.72280.50620.93300.96250.5833No cost0.75780.36370.82700.88110.7379Decision tree0.83440.75760.97300.84920.7091Easy ensemble0.94540.68760.94900.99100.9455SMOTEBoost0.90650.74520.96600.93190.8470SMOTEBagging0.90290.81300.97700.99120.8318SMOTE + SVM0.37730.46810.54460.64090.5012AdaBoost0.87120.80340.97800.95390.7758pa3636Proposed0.82870.25370.78500.83070.8933Ada-SVM0.66810.38630.95300.86890.5133CS-SVM0.66540.39260.83500.94090.6333Prorated cost0.61310.31400.93700.93990.4400No cost0.68430.31370.92500.94970.5200Decision tree0.68800.53660.97600.74150.4933Easy ensemble0.88670.36110.91800.97370.8800SMOTEBoost0.90070.75190.98500.93410.8400SMOTEBagging0.75900.58220.97600.94890.6000SMOTE + SVM0.74460.72040.76570.88610.6609AdaBoost0.79190.65370.98300.93540.6533pa4040Proposed0.81300.44210.95400.90450.7000Ada-SVM0.68500.45780.97000.91260.4900CS-SVM0.50730.12740.56300.85210.6400Prorated cost0.82310.44300.95700.95050.7500No cost0.69890.29090.93700.84050.5400Decision tree0.70790.55110.98100.76090.5300Easy ensemble0.86880.30420.90400.96080.8400SMOTEBoost0.75040.41290.95900.88090.6000SMOTEBagging0.62230.44620.97600.97700.4140SMOTE + SVM0.66960.63960.70290.79050.5237AdaBoost0.62860.50910.98200.83680.4200pa5050Proposed0.87440.32860.91300.93880.8500Ada-SVM0.67050.28120.93300.67490.5500CS-SVM0.78960.37930.90700.91260.7333Prorated cost0.49400.28190.79800.94860.4500No cost0.80950.26330.87500.94030.8000Decision tree0.72320.62760.98800.77350.5500Easy ensemble0.82760.23960.90200.95030.8000SMOTEBoost0.69370.47150.97900.94460.5167SMOTEBagging0.76030.65000.98900.95790.6167SMOTE + SVM0.63150.57220.72020.85140.4941AdaBoost0.78100.65950.98800.96670.6333The bold values represent the parts of the tests that are significant.The higher the IR of the dataset, the worse the classification performance tends to be for all methods. Our method does not perform as well on the second group datasets as first, although with the same IR as first. Overall, the classical Easy Ensemble and SMOTEBagging algorithms show a series of better performances on the Pageblock dataset. This also implies that our approach seems to be more effective with small datasets. However, we found that the recall value of the proposed model presented in Table7 is significantly better than accuracy-oriented algorithms such as No-Cost SVM, decision tree, and AdaBoost. It also outperforms the recall values of cost-sensitive classifiers such as Ada-SVM, prorated, and SMOTE + SVM. The proposed model does not have statistically significant differences in recall values with CS-SVM, Easy Ensemble, SMOTEBoost, and SMOTEBagging state-of-the-art algorithms.Table 7
P values for recall performance measures in the high IR of the Pageblock dataset.
Indicators/ModelAda-SVMCS-SVMProrated costNo costDecision treeEasy ensembleSMOTEBoostSMOTEBaggingSMOTE + SVMAdaBoostRecallT0.0000.0550.0160.0010.0230.1100.3940.1700.0000.060Wilcox0.0040.0970.0180.0060.0300.1250.5650.2500.0020.035The bold values represent the parts of the tests that are significant.The proposed, Ada-SVM, CS-SVM, prorated cost, no cost, decision tree, easy ensemble, SMOTEBoost, SMOTEBagging, SMOTE + SVM, and AdaBoost algorithms used to classify the Ecoli series dataset take approximately 100 s, 100 s, 30 s, 30 s, 5 s, 0.03 s, 50 s, 3 s, 3 s, 20 s, and 1 s, respectively. The time required for the Pageblock dataset of size 1000 is 1700 s, 1700 s, 200 s, 200 s, 40 s, 6 s, 160 s, 7 s, 7 s, 120 s, and 3 s.


## 6. Discussion and Conclusion

Unlike other classification methods that only assign different costs to different categories to achieve cost sensitivity, the proposed model built a process that enabled SVM to self-adaptively update the cost value of each sample by integrations. The misclassified positive samples were assigned higher cost values, while other misclassified negative and correctly classified instances were given lower cost items to decrease their effectiveness in training. The optimization of each base classifier can be reached by these automatically updated cost values, which are following the selfadaptively weight vector that was decided by our new weighting mechanism.Through theoretical justifications and empirical studies, the proposed approach had higher recall than others, and there were no differences with other classification measures at the 0.05 level. When dealing with imbalanced datasets, the findings demonstrated that the suggested method outperformed alternative methods statistically significantly. Through extensive experiments on different IR datasets, our method guarantees good results on the classification of a few classes on both high and low IR datasets. In some datasets, the mean recall metric of the proposed method could be one after five-fold cross-validation, while a set of other metrics could be maintained at an average level. We also found that our model underperformed in terms of overall accuracy compared with that of some models. This phenomenon stems from the purpose that we chase, our need is for higher recall rather than overall accuracy. This is momentous for the practical issue of reducing the identification overhead when working with a small number of classes. Because our method can achieve a recall value of 1 in some datasets, this good feature can be an effective aid in practical work, reducing a large amount of burden for manual work in identifying minorities, such as medical diagnosis. This is when the overall accuracy becomes less important than recall.The advantage ofP-mean in assessing the classification impact of a skewed dataset is not evident in this study; however, P-mean can be used to appraise the cost-sensitive classifier. It is worth investigating whether the advantage of this assessment metric can be proved in additional experimental instances. Furthermore, we analyzed and confirmed the results of previous studies about class structure and imbalance ratio. They indeed can have a serious impact on classification performance. The classification effect of all classifiers would be reduced when the sample overlap is excessively high; however, our method can also have the best recall performance among many classifiers. As an important reason for the classification disaster of the imbalanced dataset, the structure of classes can be studied in more depth in future work. The high IR would bring a bad classification performance.Our model has its limitations. For instance, it works better on small datasets; this may be attributed to the fact that SVM, which is the base classifier of our proposed model, is more suitable for classification problems on small datasets. As regards the longer time that our algorithm needs for classification as compared to others, if we continue to update the program code in the future without changing the model, the time computational complexity of our method will be considerably reduced. Substantial research can be conducted in the future, including parameter evaluation and improvement. The impact of kernel functions on imbalanced classification or imbalanced multiple classification problems is also worth investigating. Because of its enormous application potential, this challenging topic will continue to receive extensive attention.



---

*Source: 4860536-2023-01-19.xml*