# A Statistical Approach to Cost-Sensitive AdaBoost for Imbalanced Data Classification

Authors: Honghan Bei; Yajie Wang; Zhaonuo Ren; Shuo Jiang; Keran Li; Wenyang Wang
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

To address the imbalanced data problem in classification, the studies of the combination of AdaBoost, short for “Adaptive Boosting,” and cost-sensitive learning have shown convincing results in the literature. The cost-sensitive AdaBoost algorithms are practical since the “boosting” property in AdaBoost can iteratively enhance the small class of the cost-sensitive learning to solve the imbalanced data issue. However, the most available cost-sensitive AdaBoost algorithms are heuristic approaches, which are improved from the standard AdaBoost algorithm by cost-sensitively adjusting the voting weight parameters of weak classifiers or the sample updating weight parameters without strict theoretic proof. The algorithms are appended the cost-sensitive factors to focus on the high-cost and small-class samples, but they have no procedures to show the best place to add the cost factors and the cost factor value set. To complete the cost-sensitive AdaBoost algorithms’ framework, the present article has two main contributions. First, we summarize the popular cost-sensitive boosting algorithms in the literature and propose a generally comprehensive form. We name our specific one, the “AdaImC algorithm,” which is typically appliable to solve the imbalanced data classification problem with theoretic proof. Second, a statistical approach to prove the AdaImC algorithm is proposed to verify the inner relationship between the cost parameters. We show that our proposed algorithm in the machine learning field is identical to the Product of Experts (PoE) model in the statistics field. Besides, a way to determine the cost parameter value by the statistical analysis is introduced. Several numeric studies are listed finally to support our proposed algorithm.


---

## Body

## 1. Introduction

Classification is one of the most important tasks in the machine learning and statistical field. LetS=x1,y1,x2,y2,…,xn,yn be a binary training dataset for a classification problem, where xi∈X⊆Rl, in which X is a subset of l-dimensional vector space, and the response yi∈−1,1 indicating two classes. Define S+=x,y∈S:y=1 is the positive or minority class and S−=x,y∈S:y=−1 is the negative or majority class. The class types {minority, majority} and {positive, negative} are exchangeably described by S+ and S− in this article when we talk about the imbalanced data problem. S+ is the small class, but it is the interest of our research. Let A denote the number of the elements in a set A. Define Np=S+ and Nn=S−, the number of samples in positive and negative class, respectively.In classification, the imbalanced data problem is the most challenging one. The imbalanced class situation frequently appears in many fields, such as financial fraud detection [1], text recognition [2], sentiment classification [3], web classification [4], medical research [5], and oil exploration [6]. Under a balanced situation, Nn≈Np, many algorithms are available for classification, such as linear classifiers including logistic regression and naive Bayes classifier, support vector machines, decision trees, boosted trees, and random forest [7]. Although the above algorithms still could obtain an effective global accuracy rate in the imbalanced situation, the classification accuracy rate of S+ could be low, which is the most interesting class. Typically, if Nn≫Np, this imbalanced condition in the different classes is called the imbalanced data problem. There is no strict definition for imbalanced data since some other influential factors such as small sample size and separability [8] would make the data “imbalanced” except the skewed distribution in the data. This vague definition requires that we not just focus on the data skewness if we are designing an imbalanced data classification algorithm and some comprehensive solutions to eliminate the imbalanced impact needed to be involved in the algorithm. In this paper, only the binary imbalanced class problem is studied. Multiple-class problems can be solved based on the one-versus-all scheme [9].Two approaches [10, 11] to the class imbalanced problems, data-level and algorithm-level methods, are typically used for classification. The former is a preprocessing data method [12–14]. The basic idea is to preprocess the data to eliminate the imbalanced situation before the training process. On the other hand, the algorithm-level methods focus on training the classifier directly with the raw data. This approach mainly includes cost-sensitive learning [15] and ensemble schemes.Cost-sensitive learning is a kind of object-oriented method. The common phenomenon is that many algorithms do not solve a real problem, although they have good theoretical properties. The reason is that external factors such as the cost of misclassification, the benefit of correct classification, and the efficiency substantially impact an algorithm’s performance in the specific problems. At the present stage, most researchers in the machine learning field focus on designing and improving the structure of algorithms and ignore the impact of the external factor for an algorithm in the application. Typically, adjusting the existing algorithms by adding the external factor and importing the new evaluation measures based on the external factor improves the algorithms to make them more applicable in real problems. The cost-sensitive learning method considers the external factor by bringing in the different misclassification costs in the different classes to adjust the existing algorithms. For an imbalanced problem, by setting a higher cost for the misclassification inS+ to force the classifier to skew to S+, the accuracy rate of S+ would obtain improvement. Broadly, cost-sensitive learning is categorized into two. The first is to design the classifiers directly with cost-sensitive parameters, which is called the direct method. The most famous ones are ICET [16] and cost-sensitive decision trees [17]. The other is to design a classifier that converts any existing cost-insensitive classifiers into cost-sensitive ones, which is called the cost-sensitive metalearning method. It can be further categorized into thresholding and sampling. The famous thresholding methods are MetaCost [18], cost-sensitive naive Bayes [19], and empirical thresholding [17]. The famous sampling cost-sensitive metalearning methods are Costing [20] and Weighting [21]. The above seven traditional cost-sensitive classification algorithms are conducted in the numeric section so as to compare with the proposed ones.Ensemble schemes include boosting [22–24] and bagging [25]. The boosting method is based on the theorem of Equivalence of strong and weak learnability [26]. The AdaBoost algorithm is one of the most popular boosting methods. AdaBoost, the abbreviation of Adaptive Boosting, is a machine learning algorithm formulated by Yoav Freund and Robert Schapire [27]. AdaBoost combines a series of weak classifiers and builds the final strong one to enhance the final classifier’s performance. When the weak classifiers are boosted, they have the vote weights related to every weak classifier’s accuracy rate. After a weak classifier is added, the data weight for the next weak classifier’s training is also readjusted to give a higher weight for the misclassified instances. Thus, future weak classifiers focus on the instances that previous weak classifiers misclassified.Cost-sensitive learning and AdaBoost methods are both powerful and efficient for data classification. The combination of cost-sensitive learning and AdaBoost should provide a more potent solution. We call this kind of method “cost-sensitive AdaBoost.” The basic idea is to add a cost factor to each of the weak classifiers and then boost them to build the final classifier based on the boosting method in the AdaBoost algorithm. The cost factor used here comes from the empirical rule in the specific problem and makes the algorithms more applicable if we import the cost-related evaluation measure. The cost factor can also help every weak classifier focus on the high-cost class to eliminate the imbalance. The cost-sensitive AdaBoost can make the algorithms more applicable by considering the cost of misclassification and improve the classification accuracy rate ofS+ by adding high cost to the minority class [28].The structure of this paper is as follows. Section2 reviews the AdaBoost and cost-sensitive learning algorithms. The combinations of these two algorithms in the literature are stated, and a comprehensively general form is proposed. In Section 3, we proposed our model, AdaImC Algorithm, with the theoretic proofs. A statistical approach to the AdaImC algorithm is given in Section 4. The inner relationship between the parameters in our model is shown. The results of the numeric studies with the simulated datasets and real datasets are listed in Section 5.


## 2. AdaBoost and Cost-Sensitive Learning Algorithms

### 2.1. Classical AdaBoost

A review of the classical AdaBoost is stated in Algorithm1.Algorithm 1: The classical AdaBoost [27, 29].
Input: the training data: x1,y1,x2,y2,…,xn,yn.(0)
Choose the number of weak classifiersT, let t=1, and initialize the sample weight Dti=1/n, i=1,…,n;(1)
UnderDti, find the weak classifier htx∈−1,1 minimizing the weighted error rate ϵt=∑htxi≠yiDti;(2)
Calculate the weak classifier weight:αt=1/2ln1−ϵt/ϵt;(3)
UpdateDt+1i=Dtiexp−αtyihtxi/∑j=1nDt+1j,i=1,2,…,n;(4)
Ift=T, stop the iteration, else let t=t+1 and return to Step 1.Output: the final strong classifier is Hx=sign∑t=1Tαthtx.A weak classifierhtx presents a map from X to R, and αt is the corresponding vote weight parameter. Note that αt in Step 2 is chosen based on minimizing the exponential loss function [29], which implies that the more accurate a weak classifier is, the larger vote weight it has. Step 3 presents the data weight, Dti. If instant xi,yi is misclassified by htx, then Dt+1i>Dti in ht+1x. On the other hand, if instant xi,yi is classified correctly by htx, then Dt+1i≤Dti in ht+1x inversely [30]. The following is shown in [29].Lemma 1.
Whenhtx has a range of −1,1,(1)1ni:hxi≠yi≤∏tZt,where a is an indicator, a=1 if a is true and a=0 if a is false, and Zt=∑i=1nDtiexp−αtyihtxi,t=1,2,…,T.The left formula of (1) is the empirical error rate, and the right formula is the corresponding upper boundary. Minimizing Zt on each iteration to decrease the error rate, we obtain(2)αt=12ln1−ϵtϵt,where ϵt=∑htxi≠yiDti. The training error should be less than 0.5, the random guessing error rate, that is,(3)∑htxi=yiDti>∑htxi≠yiDti.This also induces that allαt should be positive.Exponential Loss Function is used in AdaBoost to produceDti. The Exponential Loss Function is defined as(4)Ly,fx=exp−yfx,where(5)fx=∑t=1Tαthtx.On each iteration of training the weak classifier, one must solve(6)αt,ht=argminαt,ht∑i=1nexp−yift−1xi+αthxi=argminαt,ht∑i=1nDtiexp−αtyihxi,where Dti=exp−yift−1xi. For any fixed positive value of αt, (6) can be rewritten as(7)ht=argminht∑i:yi≠htxinDti=argminhtϵt.Whenhtx is determined by (7), αt can be fixed in (2):(8)Dt+1i∝Dtiexp−αtyihtxi,i=1,2,…,n.


### 2.2. Cost-Sensitive Learning

Cost-sensitive learning is an essential method in the algorithms approach to solving the imbalanced data classification problem. Cost-sensitive learning brings different costs for different classes when misclassification happens [17, 31, 32]. It is more “needs satisfying” in real problems. Generally, classification methods in the machine learning field focus on finding the classifier with a minimum error rate. However, the cost-sensitive learning method attempts to obtain a classifier with the minimum misclassified cost. The primary difference between the cost-sensitive learning method and other algorithms is the evaluation criteria. The confusion matrix is given in Table 1.Table 1
Confusion matrix.
PredictionRow totalPositiveNegativeTruthPositiveTP = true positive rateFN = false negative ratePNegativeFP = false positive rateTN = true negative rateNColumn totalP′N′1The traditional cost-insensitive classifiers focus on minimizing the global error rate,ϵ=FN+FP, on obtaining a higher classification accuracy rate, but there is no restriction or requirement for the percentage of FN and FP in ϵ. From the cost-sensitive perspective, cost-insensitive classifiers assume that the cost factor of false-negative prediction and false-positive prediction is the same. In real problems, this assumption does not usually hold. For example, a bank would use machine learning algorithms to decide if accept a loan request. There are two evaluation criteria in this biclass classification problem:(a)
Classification result has the minimum misclassified error rate(b)
Classification result has the minimum misclassified costWhen all the false decisions have the same cost,(a) and (b) are equal. However, FP’s cost, which does not accept qualified customers’ requests, is lower than FN’s cost that accepts nonqualified customers’ requests. Table 2 shows a Cost Confusion Matrix describing the cost under the different situations in Table 1. In cost-sensitive learning, we are seeking a classifier minimizing c=FP∗c1,0+FN∗c0,1. Cost-sensitive learning is practical for many other real problems, such as hazardous act detection and cancer diagnosis. The formal definition of Cost Confusion Matrix is as follows.Table 2
Cost confusion matrix.
PredictionPositiveNegativeTruthPositivec1,1c0,1Negativec1,0c0,0Definition 1.
LetCm×m be the cost matrix for a given training data, where m is the number of classes. Every nonnegative element ci,j in Cm×m means the cost of predicting class i, but the true class is j, here i,j=1,2,3,…,m.Remark 1.
(a)
Cost confusion matrix needs to be determined before the training process. Most times, it is provided by the data collector based on the empirical rule.(b)
Cost confusion matrix needs to obey the reasonableness condition, which means that the cost of misclassification should be higher than the cost of correct classification,i.e., ci,j>cj,j,i≠j for i,j=1,2,…,m. Usually, people treat cj,j=0 and ci,j>0,i≠j for simplification.(c)
In an imbalanced biclass problem, the positive class is the primary interest, so that the cost ofFN should be higher than the cost of FP, i.e., cFN>cFP. By setting this cost, the algorithm focuses on the positive class in the training process.Cost-sensitive learning is equivalent to the optimization problem applied to a specific loss function.Theorem 1.
Letci,j defined in Definition 1 be the cost of predicting class i when the true class is j. If i=j, the prediction is correct; if i≠j, the prediction is incorrect, where i,j=1,2,3,…,m. For a cost-sensitive learning algorithm A, the optimal prediction for an instant in class i is to minimize(9)Lx,i=∑jPj|xci,j,where x,i means the prediction class of x is i and Pj|x means the probability of x belongs to class j under the algorithm A.Corollary 1.
In the biclass case, Theorem1 can be written as(10)Pj=−1|xc1,−1+Pj=−1|xc1,1≤Pj=−1|xc−1,−1+Pj=−1|xc−1,1,which is equivalent to(11)1−pc1,−1+pc1,1≤1−pc−1,−1+pc−1,1,where p=Pj=1|x.Corollary 2.
Let the threshold to make the optimal decision isp∗, which has the property of(12)1−p∗c1,−1+p∗c1,1=1−p∗c−1,−1+p∗c−1,1,and we have(13)p∗=c1,−1−c−1,−1c1,−1−c−1,−1+c−1,1−c1,1.Under the reasonableness condition, the optimal prediction is the positive class (class 1) if and only ifp≥p∗.The description of the general cost-sensitive learning algorithm is given in Algorithm2.Algorithm 2: The biclass cost-sensitive learning algorithm [8].
Input: the training data: x1,y1,cy1,x2,y2,cy2,…,xn,yn,cyn. Every xi is an l−dimensional variable in X, yi is a class label in Y=−1,1, and cyi⊆0,+∞ is the associating misclassification cost for every class;(1)
Define the data weightD based on X×Y and the cost matrix C2×2 based on Definition 1;(2)
Under thisD and C2×2, find the classifier htx minimizing EX,Y∼Dcosty,yhx;Output: the final classifier is hx.Although some cost-sensitive learning methods are starting from building a new algorithm structure based on available cost information, such as the cost-sensitive decision tree pruning method [33], most of the existing cost-sensitive learning methods are adjusted from the cost-insensitive algorithms by adding the cost information [34]. This approach transfers the minimizing global error algorithm to the minimizing expected cost algorithm, and it is highly optimized for general use, including the rescaling method and reweighted method. Typically, the rescaling method is a resampling approach that preprocesses the training data based on the cost information to change the data distribution before the training process. The classifier will be biased to the cost-sensitive class and improve its classification accuracy rate. Rescaling is another data-level method but considering the cost-sensitive information [1, 35–37]. On the other hand, the reweighted approach, including the cost-sensitive ensemble learning method, focuses on adjusting the cost-insensitive algorithms. It brings cost information to obtain cost-sensitive property and then boosts a series of cost-sensitive weak classifiers to obtain a strong one. MetaCost [18] and Proportion-Cost [20] are two other interesting reweighted approaches in cost-sensitive learning. Figure 1 gives a family tree to illustrate the relationship between all the classifiers we mentioned above.Figure 1
Various methods for classification.


### 2.3. Cost-Sensitive AdaBoost

We see several kinds of cost-sensitive AdaBoost algorithms in the literature, such as AdaCost series algorithms [38], AdaC series algorithms [8], CBS series algorithms [39], and Asymmetric Boosting algorithm [40]. These algorithms are the most popular ones, but they are heuristic without strict theoretic proof. Their basic idea is to add cost factors to the vote weights of the weak classifiers αt or the data weights Dt in AdaBoost. Table 3 summarizes adjusting the classical AdaBoost to obtain the above existing cost-sensitive AdaBoost algorithms.Table 3
Summary of cost-sensitive AdaBoost variants showing how they modifyDt and αt.
AlgorithmDt+1∝×DtαtParameter descriptionClassical AdaBoost [27]e−yiαthtxi1/2log∑i:htxi=yiDti/∑i:htxi≠yiDtiAdaCost [38]e−βtiyiαthtxi1/2log1+γt/1−γtD1i=cyi∈0,1γt=∑i=1nDtiyihtxiβtiβti=1+cyi/2,ifhtxi≠yi,1−cyi/2,ifhtxi=yi.CBS0γti1/2log∑i:htxi=yiDti/∑i:htxi≠yiDtiD1i=cyiCBS1γtie−yihtxi1/2log∑i:htxi=yiDti/∑i:htxi≠yiDtiγti=cyi,ifhtxi≠yi,1,ifhtxi=yi.CBS2 [39]γtie−yiαthtxi1/2log∑i:htxi=yiDti/∑i:htxi≠yiDtiAsymAda [40]cyi1/Me−yiαthtxi1/2log∑i:htxi=yiDticyi1/M/∑i:htxi≠yiDticyi1/MD1i=cyi1/M (M is fixed)AdaC1e−cyiyiαthtxi1/2log1+∑i:htxi=yiDticyi−∑i:htxi≠yiDticyi/1−∑i:htxi=yiDticyi+∑i:htxi≠yiDticyiD1i=cyiAdaC2cyie−yiαthtxi1/2log∑i:htxi=yiDticyi/∑i:htxi≠yiDticyiD1i=cyiAdaC3 [8]cyie−cyiyiαthtxi1/2log∑i=1nDticyi+∑i:htxi=yiDticyi2−∑i:htxi≠yiDticyi2/∑i=1nDticyi−∑i:htxi=yiDticyi2+∑i:htxi≠yiDticyi2D1i=cyiSome other cost-sensitive AdaBoost algorithms, such as CGAda [41] and CSAda [42], are not widespread, which we do not consider in this article. Some researchers have done excellent work to analyze the existing cost-sensitive AdaBoost algorithms in Table 3, which are used frequently in the machine learning field. Cao et al. [14] listed the weakness of some popular cost-sensitive AdaBoost algorithms, including those in Table 3, and concluded that most of them are heuristic and lack the theoretical analysis. Nikolaou et al. [43] gave a summary of them and analyzed them from a mathematical perspective. One of the most critical properties of boosting is the robustness of the resulting, which means that the final classifier tends to continue reducing the test error even after perfect classification is reached on the training set. The heuristic adjustment of AdaBoost would lose this property in Boosting.The adjusted point in the AdaCost is the cost functionβti. In the high-cost class, βti is low in the correctly classified situations and is high in the misclassified situations. Nevertheless, AdaCost did not consider the change in αt. In order to keep the reasonability of αtαt>0, every weak classifier needs to satisfy ∑i=1nDiβhxi=yii>∑i=1nDiβhxi≠yii. This restriction requires a very high accuracy rate for the weak classifiers. In an imbalanced data classification problem, people can often not obtain the powerful weak classifiers for boosting. Because of the limit of efficiency, AdaCost does not perform well in imbalanced data classification.CSB is a further study based on AdaCost. Ting and Ming [39] compared CSB series algorithms and AdaCost algorithm to conclude that CSB performs better than AdaCost. However, the apparent shortcomings of CSB are that it only adjusts the data weight Dt and does not adjust the corresponding weak classifier vote weight αt. This heuristic adjusting deviates from the original bboosting property in AdaBoost.AsymAda, short for Asymmetric Boosting, applies the gradient descent optimization method to the cost-sensitive loss functionLx,y,hx=Ex,y∼DIy=1e−cyiyhx+Iy=−1e−cyiyhx. Nevertheless, people have to use a numerical method such as the bisection search to find αt [42]. This would increase the complexity of algorithms and require more time to train the model.AdaC series algorithms consider three different positions of the cost factor inDt and then infer αt based on Dt. AdaC series algorithms have strict theoretical analysis to support the algorithm’s reasonability following the proofs in [29]. The authors concluded that AC2 is superior to its rivals, and several numerical studies support this conclusion. Our proposed model starts with AdaC3. In AdaC3, Dt+1i∝cyiecyiyiαthtxi×Dti, two positions contain the same cost factor cyi. cyi outside of the exponential is the adjustment for Dt, but cyi inside of the exponential is the adjustment for αt. It is unnecessary to set the same cost factor value on these two positions. Based on this, the AdaC3 can be expanded to a generally comprehensive formation of cost-sensitive AdaBoost in the following.Definition 2.
Given the training datax1,y1,cy1,x2,y2,cy2,…,xn,yn,cyn, where xi is the feature variable, yi=−1,1 is the class variable, and cyi is the misclassified cost for every instant. Let C1icyi is a function of cyi and C2tcyi,Dti is a function of cyi and Dti. The cost-sensitive adjustment of Dt in AdaBoost is(14)Dt+1i∝C1icyieC2tcyi,Dtiyihtxi×Dti,This comprehensively general formation includes all the algorithms in Table3. The general form of the cost-sensitive AdaBoost algorithm is given in Algorithm 3.Algorithm 3: The general cost-sensitive AdaBoost.
Input: the training data: x1,y1,cy1,x2,y2,cy2,…,xn,yn,cyn, where xi is the feature variable, yi∈−1,1 is the class variable, and cyi is the misclassification cost for every instant.(0)
Choose the number of weak classifiersT, let t=1, and initialize the sample weight Dti=cyi, i=1,…,n;(1)
Under thisDti, find the weak classifier htx∈−1,1 minimizing the weighted error rate ϵt=∑htxi≠yiDti;(2)
Chooseαt∈R+;(3)
UpdateDt+1i∝C1icyieC2tcyi,Dtiyihtxi×Dti,i=1,2,…,n;(4)
Ift=T, stop the iteration, else let t=t+1 and return to Step 1.Output: the final strong classifier is Hx=sign∑t=1Tαthtx.This article aims to find specific formations ofC1icyi and C2tcyi,Dti for designing a new cost-sensitive AdaBoost for imbalanced data classification. The proposed model is biased to the positive class by the cost information and obeys the boosting property.


### 2.4. Evaluation Measures

In the cost-sensitive learning method, the most popular evaluation measure is to compare the cost of different algorithms. Ting [39] used two cost-related measures to evaluate the performance of algorithms:(1)
The total cost of misclassification made by the classifier on the test data, i.e.,∑i:yi≠hxicyi(2)
The number of high-cost errors is the number of misclassified samples in the high-cost classOther evaluation measures, such as the average misclassification cost for a series of cost values, are also used frequently. For example, letc−1,1=1 and c1,−1=0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1. The average misclassified cost for all these ten pairs of values is the evaluation measure.Because our interest is the imbalanced data classification, some people only focus on the accuracy rate and use the traditional evaluation measures of classification for cost-sensitive AdaBoost such asF−1Measure, Recall, and Precision [8]. This is reasonable when our interest is the accuracy rate of the positive class, and the cost is only a tool to neutralize the imbalanced situation. Cost-sensitive adjustment in AdaBoost is just a tool to force the algorithm to focus on the positive class. Based on Confusion Matrix 1, we can define some basic evaluation measures as follows:Global Accuracy=TP+TNSensitivity=Recall=TP/PSpecificity=TN/NPrecision=TP/P′F−1Measure=2×Precision×Sensitivity/Precision+SensitivitySensitivityRecall and Precision are the measures for accessing the classification performance of S+. F−1Measure was suggested in [44] to integrate the Sensitive and Precision as an average, representing a harmonic mean between Sensitive and Precision. The harmonic mean of two numbers is closer to the smaller one. So, a high F−1Measure can ensure that the Sensitive and Precision are both high.This paper will consider the traditional evaluation measures for the real data studies and cost-related evaluation measures for the simulation data studies.


## 2.1. Classical AdaBoost

A review of the classical AdaBoost is stated in Algorithm1.Algorithm 1: The classical AdaBoost [27, 29].
Input: the training data: x1,y1,x2,y2,…,xn,yn.(0)
Choose the number of weak classifiersT, let t=1, and initialize the sample weight Dti=1/n, i=1,…,n;(1)
UnderDti, find the weak classifier htx∈−1,1 minimizing the weighted error rate ϵt=∑htxi≠yiDti;(2)
Calculate the weak classifier weight:αt=1/2ln1−ϵt/ϵt;(3)
UpdateDt+1i=Dtiexp−αtyihtxi/∑j=1nDt+1j,i=1,2,…,n;(4)
Ift=T, stop the iteration, else let t=t+1 and return to Step 1.Output: the final strong classifier is Hx=sign∑t=1Tαthtx.A weak classifierhtx presents a map from X to R, and αt is the corresponding vote weight parameter. Note that αt in Step 2 is chosen based on minimizing the exponential loss function [29], which implies that the more accurate a weak classifier is, the larger vote weight it has. Step 3 presents the data weight, Dti. If instant xi,yi is misclassified by htx, then Dt+1i>Dti in ht+1x. On the other hand, if instant xi,yi is classified correctly by htx, then Dt+1i≤Dti in ht+1x inversely [30]. The following is shown in [29].Lemma 1.
Whenhtx has a range of −1,1,(1)1ni:hxi≠yi≤∏tZt,where a is an indicator, a=1 if a is true and a=0 if a is false, and Zt=∑i=1nDtiexp−αtyihtxi,t=1,2,…,T.The left formula of (1) is the empirical error rate, and the right formula is the corresponding upper boundary. Minimizing Zt on each iteration to decrease the error rate, we obtain(2)αt=12ln1−ϵtϵt,where ϵt=∑htxi≠yiDti. The training error should be less than 0.5, the random guessing error rate, that is,(3)∑htxi=yiDti>∑htxi≠yiDti.This also induces that allαt should be positive.Exponential Loss Function is used in AdaBoost to produceDti. The Exponential Loss Function is defined as(4)Ly,fx=exp−yfx,where(5)fx=∑t=1Tαthtx.On each iteration of training the weak classifier, one must solve(6)αt,ht=argminαt,ht∑i=1nexp−yift−1xi+αthxi=argminαt,ht∑i=1nDtiexp−αtyihxi,where Dti=exp−yift−1xi. For any fixed positive value of αt, (6) can be rewritten as(7)ht=argminht∑i:yi≠htxinDti=argminhtϵt.Whenhtx is determined by (7), αt can be fixed in (2):(8)Dt+1i∝Dtiexp−αtyihtxi,i=1,2,…,n.


## 2.2. Cost-Sensitive Learning

Cost-sensitive learning is an essential method in the algorithms approach to solving the imbalanced data classification problem. Cost-sensitive learning brings different costs for different classes when misclassification happens [17, 31, 32]. It is more “needs satisfying” in real problems. Generally, classification methods in the machine learning field focus on finding the classifier with a minimum error rate. However, the cost-sensitive learning method attempts to obtain a classifier with the minimum misclassified cost. The primary difference between the cost-sensitive learning method and other algorithms is the evaluation criteria. The confusion matrix is given in Table 1.Table 1
Confusion matrix.
PredictionRow totalPositiveNegativeTruthPositiveTP = true positive rateFN = false negative ratePNegativeFP = false positive rateTN = true negative rateNColumn totalP′N′1The traditional cost-insensitive classifiers focus on minimizing the global error rate,ϵ=FN+FP, on obtaining a higher classification accuracy rate, but there is no restriction or requirement for the percentage of FN and FP in ϵ. From the cost-sensitive perspective, cost-insensitive classifiers assume that the cost factor of false-negative prediction and false-positive prediction is the same. In real problems, this assumption does not usually hold. For example, a bank would use machine learning algorithms to decide if accept a loan request. There are two evaluation criteria in this biclass classification problem:(a)
Classification result has the minimum misclassified error rate(b)
Classification result has the minimum misclassified costWhen all the false decisions have the same cost,(a) and (b) are equal. However, FP’s cost, which does not accept qualified customers’ requests, is lower than FN’s cost that accepts nonqualified customers’ requests. Table 2 shows a Cost Confusion Matrix describing the cost under the different situations in Table 1. In cost-sensitive learning, we are seeking a classifier minimizing c=FP∗c1,0+FN∗c0,1. Cost-sensitive learning is practical for many other real problems, such as hazardous act detection and cancer diagnosis. The formal definition of Cost Confusion Matrix is as follows.Table 2
Cost confusion matrix.
PredictionPositiveNegativeTruthPositivec1,1c0,1Negativec1,0c0,0Definition 1.
LetCm×m be the cost matrix for a given training data, where m is the number of classes. Every nonnegative element ci,j in Cm×m means the cost of predicting class i, but the true class is j, here i,j=1,2,3,…,m.Remark 1.
(a)
Cost confusion matrix needs to be determined before the training process. Most times, it is provided by the data collector based on the empirical rule.(b)
Cost confusion matrix needs to obey the reasonableness condition, which means that the cost of misclassification should be higher than the cost of correct classification,i.e., ci,j>cj,j,i≠j for i,j=1,2,…,m. Usually, people treat cj,j=0 and ci,j>0,i≠j for simplification.(c)
In an imbalanced biclass problem, the positive class is the primary interest, so that the cost ofFN should be higher than the cost of FP, i.e., cFN>cFP. By setting this cost, the algorithm focuses on the positive class in the training process.Cost-sensitive learning is equivalent to the optimization problem applied to a specific loss function.Theorem 1.
Letci,j defined in Definition 1 be the cost of predicting class i when the true class is j. If i=j, the prediction is correct; if i≠j, the prediction is incorrect, where i,j=1,2,3,…,m. For a cost-sensitive learning algorithm A, the optimal prediction for an instant in class i is to minimize(9)Lx,i=∑jPj|xci,j,where x,i means the prediction class of x is i and Pj|x means the probability of x belongs to class j under the algorithm A.Corollary 1.
In the biclass case, Theorem1 can be written as(10)Pj=−1|xc1,−1+Pj=−1|xc1,1≤Pj=−1|xc−1,−1+Pj=−1|xc−1,1,which is equivalent to(11)1−pc1,−1+pc1,1≤1−pc−1,−1+pc−1,1,where p=Pj=1|x.Corollary 2.
Let the threshold to make the optimal decision isp∗, which has the property of(12)1−p∗c1,−1+p∗c1,1=1−p∗c−1,−1+p∗c−1,1,and we have(13)p∗=c1,−1−c−1,−1c1,−1−c−1,−1+c−1,1−c1,1.Under the reasonableness condition, the optimal prediction is the positive class (class 1) if and only ifp≥p∗.The description of the general cost-sensitive learning algorithm is given in Algorithm2.Algorithm 2: The biclass cost-sensitive learning algorithm [8].
Input: the training data: x1,y1,cy1,x2,y2,cy2,…,xn,yn,cyn. Every xi is an l−dimensional variable in X, yi is a class label in Y=−1,1, and cyi⊆0,+∞ is the associating misclassification cost for every class;(1)
Define the data weightD based on X×Y and the cost matrix C2×2 based on Definition 1;(2)
Under thisD and C2×2, find the classifier htx minimizing EX,Y∼Dcosty,yhx;Output: the final classifier is hx.Although some cost-sensitive learning methods are starting from building a new algorithm structure based on available cost information, such as the cost-sensitive decision tree pruning method [33], most of the existing cost-sensitive learning methods are adjusted from the cost-insensitive algorithms by adding the cost information [34]. This approach transfers the minimizing global error algorithm to the minimizing expected cost algorithm, and it is highly optimized for general use, including the rescaling method and reweighted method. Typically, the rescaling method is a resampling approach that preprocesses the training data based on the cost information to change the data distribution before the training process. The classifier will be biased to the cost-sensitive class and improve its classification accuracy rate. Rescaling is another data-level method but considering the cost-sensitive information [1, 35–37]. On the other hand, the reweighted approach, including the cost-sensitive ensemble learning method, focuses on adjusting the cost-insensitive algorithms. It brings cost information to obtain cost-sensitive property and then boosts a series of cost-sensitive weak classifiers to obtain a strong one. MetaCost [18] and Proportion-Cost [20] are two other interesting reweighted approaches in cost-sensitive learning. Figure 1 gives a family tree to illustrate the relationship between all the classifiers we mentioned above.Figure 1
Various methods for classification.


## 2.3. Cost-Sensitive AdaBoost

We see several kinds of cost-sensitive AdaBoost algorithms in the literature, such as AdaCost series algorithms [38], AdaC series algorithms [8], CBS series algorithms [39], and Asymmetric Boosting algorithm [40]. These algorithms are the most popular ones, but they are heuristic without strict theoretic proof. Their basic idea is to add cost factors to the vote weights of the weak classifiers αt or the data weights Dt in AdaBoost. Table 3 summarizes adjusting the classical AdaBoost to obtain the above existing cost-sensitive AdaBoost algorithms.Table 3
Summary of cost-sensitive AdaBoost variants showing how they modifyDt and αt.
AlgorithmDt+1∝×DtαtParameter descriptionClassical AdaBoost [27]e−yiαthtxi1/2log∑i:htxi=yiDti/∑i:htxi≠yiDtiAdaCost [38]e−βtiyiαthtxi1/2log1+γt/1−γtD1i=cyi∈0,1γt=∑i=1nDtiyihtxiβtiβti=1+cyi/2,ifhtxi≠yi,1−cyi/2,ifhtxi=yi.CBS0γti1/2log∑i:htxi=yiDti/∑i:htxi≠yiDtiD1i=cyiCBS1γtie−yihtxi1/2log∑i:htxi=yiDti/∑i:htxi≠yiDtiγti=cyi,ifhtxi≠yi,1,ifhtxi=yi.CBS2 [39]γtie−yiαthtxi1/2log∑i:htxi=yiDti/∑i:htxi≠yiDtiAsymAda [40]cyi1/Me−yiαthtxi1/2log∑i:htxi=yiDticyi1/M/∑i:htxi≠yiDticyi1/MD1i=cyi1/M (M is fixed)AdaC1e−cyiyiαthtxi1/2log1+∑i:htxi=yiDticyi−∑i:htxi≠yiDticyi/1−∑i:htxi=yiDticyi+∑i:htxi≠yiDticyiD1i=cyiAdaC2cyie−yiαthtxi1/2log∑i:htxi=yiDticyi/∑i:htxi≠yiDticyiD1i=cyiAdaC3 [8]cyie−cyiyiαthtxi1/2log∑i=1nDticyi+∑i:htxi=yiDticyi2−∑i:htxi≠yiDticyi2/∑i=1nDticyi−∑i:htxi=yiDticyi2+∑i:htxi≠yiDticyi2D1i=cyiSome other cost-sensitive AdaBoost algorithms, such as CGAda [41] and CSAda [42], are not widespread, which we do not consider in this article. Some researchers have done excellent work to analyze the existing cost-sensitive AdaBoost algorithms in Table 3, which are used frequently in the machine learning field. Cao et al. [14] listed the weakness of some popular cost-sensitive AdaBoost algorithms, including those in Table 3, and concluded that most of them are heuristic and lack the theoretical analysis. Nikolaou et al. [43] gave a summary of them and analyzed them from a mathematical perspective. One of the most critical properties of boosting is the robustness of the resulting, which means that the final classifier tends to continue reducing the test error even after perfect classification is reached on the training set. The heuristic adjustment of AdaBoost would lose this property in Boosting.The adjusted point in the AdaCost is the cost functionβti. In the high-cost class, βti is low in the correctly classified situations and is high in the misclassified situations. Nevertheless, AdaCost did not consider the change in αt. In order to keep the reasonability of αtαt>0, every weak classifier needs to satisfy ∑i=1nDiβhxi=yii>∑i=1nDiβhxi≠yii. This restriction requires a very high accuracy rate for the weak classifiers. In an imbalanced data classification problem, people can often not obtain the powerful weak classifiers for boosting. Because of the limit of efficiency, AdaCost does not perform well in imbalanced data classification.CSB is a further study based on AdaCost. Ting and Ming [39] compared CSB series algorithms and AdaCost algorithm to conclude that CSB performs better than AdaCost. However, the apparent shortcomings of CSB are that it only adjusts the data weight Dt and does not adjust the corresponding weak classifier vote weight αt. This heuristic adjusting deviates from the original bboosting property in AdaBoost.AsymAda, short for Asymmetric Boosting, applies the gradient descent optimization method to the cost-sensitive loss functionLx,y,hx=Ex,y∼DIy=1e−cyiyhx+Iy=−1e−cyiyhx. Nevertheless, people have to use a numerical method such as the bisection search to find αt [42]. This would increase the complexity of algorithms and require more time to train the model.AdaC series algorithms consider three different positions of the cost factor inDt and then infer αt based on Dt. AdaC series algorithms have strict theoretical analysis to support the algorithm’s reasonability following the proofs in [29]. The authors concluded that AC2 is superior to its rivals, and several numerical studies support this conclusion. Our proposed model starts with AdaC3. In AdaC3, Dt+1i∝cyiecyiyiαthtxi×Dti, two positions contain the same cost factor cyi. cyi outside of the exponential is the adjustment for Dt, but cyi inside of the exponential is the adjustment for αt. It is unnecessary to set the same cost factor value on these two positions. Based on this, the AdaC3 can be expanded to a generally comprehensive formation of cost-sensitive AdaBoost in the following.Definition 2.
Given the training datax1,y1,cy1,x2,y2,cy2,…,xn,yn,cyn, where xi is the feature variable, yi=−1,1 is the class variable, and cyi is the misclassified cost for every instant. Let C1icyi is a function of cyi and C2tcyi,Dti is a function of cyi and Dti. The cost-sensitive adjustment of Dt in AdaBoost is(14)Dt+1i∝C1icyieC2tcyi,Dtiyihtxi×Dti,This comprehensively general formation includes all the algorithms in Table3. The general form of the cost-sensitive AdaBoost algorithm is given in Algorithm 3.Algorithm 3: The general cost-sensitive AdaBoost.
Input: the training data: x1,y1,cy1,x2,y2,cy2,…,xn,yn,cyn, where xi is the feature variable, yi∈−1,1 is the class variable, and cyi is the misclassification cost for every instant.(0)
Choose the number of weak classifiersT, let t=1, and initialize the sample weight Dti=cyi, i=1,…,n;(1)
Under thisDti, find the weak classifier htx∈−1,1 minimizing the weighted error rate ϵt=∑htxi≠yiDti;(2)
Chooseαt∈R+;(3)
UpdateDt+1i∝C1icyieC2tcyi,Dtiyihtxi×Dti,i=1,2,…,n;(4)
Ift=T, stop the iteration, else let t=t+1 and return to Step 1.Output: the final strong classifier is Hx=sign∑t=1Tαthtx.This article aims to find specific formations ofC1icyi and C2tcyi,Dti for designing a new cost-sensitive AdaBoost for imbalanced data classification. The proposed model is biased to the positive class by the cost information and obeys the boosting property.


## 2.4. Evaluation Measures

In the cost-sensitive learning method, the most popular evaluation measure is to compare the cost of different algorithms. Ting [39] used two cost-related measures to evaluate the performance of algorithms:(1)
The total cost of misclassification made by the classifier on the test data, i.e.,∑i:yi≠hxicyi(2)
The number of high-cost errors is the number of misclassified samples in the high-cost classOther evaluation measures, such as the average misclassification cost for a series of cost values, are also used frequently. For example, letc−1,1=1 and c1,−1=0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1. The average misclassified cost for all these ten pairs of values is the evaluation measure.Because our interest is the imbalanced data classification, some people only focus on the accuracy rate and use the traditional evaluation measures of classification for cost-sensitive AdaBoost such asF−1Measure, Recall, and Precision [8]. This is reasonable when our interest is the accuracy rate of the positive class, and the cost is only a tool to neutralize the imbalanced situation. Cost-sensitive adjustment in AdaBoost is just a tool to force the algorithm to focus on the positive class. Based on Confusion Matrix 1, we can define some basic evaluation measures as follows:Global Accuracy=TP+TNSensitivity=Recall=TP/PSpecificity=TN/NPrecision=TP/P′F−1Measure=2×Precision×Sensitivity/Precision+SensitivitySensitivityRecall and Precision are the measures for accessing the classification performance of S+. F−1Measure was suggested in [44] to integrate the Sensitive and Precision as an average, representing a harmonic mean between Sensitive and Precision. The harmonic mean of two numbers is closer to the smaller one. So, a high F−1Measure can ensure that the Sensitive and Precision are both high.This paper will consider the traditional evaluation measures for the real data studies and cost-related evaluation measures for the simulation data studies.


## 3. AdaImC Algorithm

The term AdaImC means the cost-sensitive AdaBoost algorithm designed for imbalanced data classification. Recall the general expression of the adjustment in cost-sensitive AdaBoost in Algorithm3:(15)Dt+1i∝C1icyieC2tcyi,Dtiyihtxi×Dti,where C1icyi works for adding the cost factor to the data weight update procedure and C2tcyi,Dti is an adjustment for αt.Definition 3.
The specific adjustment ofDt in AdaImC is(16)Dt+1i∝Dtic11−yi/2exp−yihtxiαt+c2,where c1∈0,1 and c2∈0,+∞.The adjustment in Definition3 is similar to the AdaC3 algorithm. The main difference is that we add a new parameter c2 to αt instead of multiplying cyi by αt. c11−yi/2 is the adjustment for the sample weight parameters. For the positive class, yi=1 and c11−yi/2=1; for the negative class, yi=−1 and c11−yi/2=c1. This adjustment enhances the weight parameters for the positive class since c1<1. The purpose of the adjustment c2 is to add effect on the voting weight parameters αt of the weak classifiers so that the outstanding weak classifiers can contribute more to the final classifier.Remark 2.
In Definition3, 0<c1<1 and c2>0.The reason of0<c1<1 is obvious because c11−yi/2=1,ifyi=1,c1,ifyi=−1. The positive class cost should be higher than the negative class, and all the cost values are nonnegative. c2>0 gives a heavier effect to αt.Theorem 2.
Assuming Algorithm1 replaces the equation in Algorithm 3, we have(17)1n∑i=1nhxi≠yi≤1c1∏t=1TZt,where a=1 if a is true and a=0 if a is false, and(18)Zt=∑i=1nDtic11−yi/2exp−yihtxiαt+c2.For the proof of Theorem2, see Appendix ATheorem2 indicates an upper boundary of the global error rate for the AdaImC algorithm. The minimum global error rate is the foundation for any other improvement for the accuracy rate of the positive class in AdaBoost. The next step is to find the minimum value of Zt and then get at under this minimum global error setting.Theorem 3.
Replace the equation in Algorithm3 with Algorithm 1 to obtain the minimum upper boundary of the global error rate. We have(19)αt=12ln∑i:htxi=yiDitc11−yi/2∑i:htxi≠yiDitc11−yi/2−c2.For the proof of Theorem3, see Appendix B.Remark 3.
We know thatαt must be positive to make the vote weight parameter reasonable. So, the restriction for c2 is(20)0<c2<12ln∑i:htxi=yiDitc11−yi/2∑i:htxi≠yiDitc11−yi/2.The construction of AdaImC algorithmis given in Algorithm4.Algorithm 4: The AdaImC algorithm.
Input: the training data: x1,y1,x2,y2,…,xn,yn, where xi is the feature variable and yi∈−1,1 is the class variable.(0)
Choose the number of weak classifiersT. Determine the constants 0<c1<1, and let t=1 to initialize the sample weight Dti=c11−yi/2; determine 0<c2<1/2ln∑i:htxi=yiDitc11−yi/2/∑i:htxi≠yiDitc11−yi/2, i=1,…,n;(1)
Under thisDti, find the weak classifier htx∈−1,1 minimizing the weighted error rate ϵt=∑htxi≠yiDti;(2)
Calculateαt=1/2ln∑i:htxi=yiDitc11−yi/2/∑i:htxi≠yiDitc11−yi/2−c2;(3)
UpdateDt+1i=c11−yi/2Dtiexp−yihtxiαt+c2/n∏t=1TZt,i=1,2,…,n,hereZt=∑i=1nDtic11−yi/2exp−yihtxi+c2αt;(4)
Ift=T, stop the iteration, else let t=t+1 and return to Step 1.Output: the final strong classifier is Hx=sign∑t=1Tαthtx.We describe the AdaImC algorithm and show the theoretic proof to support our improvement. The algorithm construction in this section follows the theorems and proofs in previous cost-sensitive AdaBoost algorithms but has no information about determining the cost parameters. The next section tries to build the AdaImC algorithm from a statistical perspective and show the deeper inner connections between the parameters.


## 4. A Statistical Approach to AdaImC Algorithm

Although there is not much theoretic proof for the cost-sensitive boosting algorithms in the literature, we have convincing statistical studies on the original AdaBoost. This section’s basic idea is that the direct proof of AdaImC algorithms is hard to locate so that we show it is identical to the Product of Expert (PoE) model [45] in statistics. All the convincing properties and proofs of the PoE model can support the theorems of the AdaImC algorithm. This section follows the typically statistical approaches [46, 47] to the original AdaBoost and shows a statistical perspective to prove our proposed AdaImC algorithm. First, we set the following assumptions.Assumption 1.
(1)
All the observations in the dataset are independent and identically distributed(2)
All the weak classifiers are independentEvery weak classifier is trained based on the data weight related to the previous weak classifiers’ performance. So, it is hard to conclude that all the weak classifiers are independent.LetX be the random independent variable, andx is the realization of X. Let Y be the random dependent variable, and y is the realization of Y and y¯ is the complement of y. We drop the random variables from all the formulas for notational convenience.Fact 1.
From Assumption1, we have(21)py|x,h1,h2,…,hJ=∏j=1Tpy|x,hj∏j=1Tpy|x,hj+∏j=1Tpy¯|x,hj,(22)py1,y2,…,yn|x1,x2,…,xn,h1x,h2x,…,hjx=∏i=1npyi|xi,h1x,h2x,…,hjx,(23)py1,y2,…,yn|x1,x2,…,xn,h1x,h2x,…,hj−1x=∏i=1npyi|xi,h1x,h2x,…,hj−1x.FromFact 1, we have(24)py|x,h1,h2,…,hj=AB+C,where(25)A=py|x,hjpy|x,h1,h2,…,hj−1,B=py|x,hjpy|x,h1,h2,…,hj−1,C=py¯|x,hjpy¯|x,h1,h2,…,hj−1.Our goal is that, with every weak classifier added, we would like the likelihood of the observed data conditioned on the classifiers to increase or remain the same, namely,(26)py1,…,yn|x1,…,xn,h1x,…,hjx≥py1,…,yn|x1,…,xn,h1x,…,hj−1x.Theorem 4.
The sufficient condition of (26) is(27)∑i=1nDij−1pyi|xi,hj≤2,where(28)Dij−1=pyi¯|xi,h1,h2,…,hj∑i=1npyi¯|xi,h1,h2,…,hj.For the proof of Theorem4, see Appendix C.Using (23), we can simplify (28) as(29)Dij=pyi¯|xi,hjpyi¯|xi,h1,…,hj−1/Qij∑i=1Npyi¯|xi,hjpyi¯|xi,h1,…,hj−1/Qij=pyi¯|xi,hjDij−1/Qij∑i=1Npyi¯|xi,hjDij−1/Qij,where Qij is the normalization term given by(30)Qij=pyi|xi,hjpyi|xi,h1,…,hj−1+pyi¯|xi,hjpyi¯|xi,h1,…,hj−1.Now, we can build an incremental learning process named the PoE algorithm (Product of Experts model) (Algorithm5).Algorithm 5: The PoE algorithm.
Input: the training data: x1,y1,x2,y2,…,xn,yn, where xi is the feature variable and yi∈−1,1 is the class variable;(0)
Choose the number of weak classifiersJ, let j=1, and initialize the sample weight Dij=1/n;(1)
Under thisDij, find the weak classifier htx∈−1,1 satisfying ∑i=1nDij/pyi|xi,hj≤2;(2)
UpdateDij+1=Dijpy¯i|xi,hj,Dij+1=Dij+1/∑inDij+1;(4)
Ifj=J, stop the iteration, else let j=j+1 and return to Step 1.Output: the final strong classifier is Hx=signpy=1|x,h1,h2,…,hJ−py=−1|x,h1,h2,…,hJ.The PoE model does not consider the weights of every weak classifier. Also, for considering the cost factor in the model, let us assume the conditional probability contributed by a weak classifier as(31)pY=y|X=x,hx=pY=y|Z=y⋅pZ=y|X=x,hx+pY=y|Z=y¯⋅pZ=y¯|X=x,hx,where Z is the random variable presenting the prediction made by the classifier. Assume(32)pY|Z=y=c11−y/2,pY=−1|Z=1=pec1,pY=1|Z=−1=pe,where c1 is the cost factor, 0<c1<1, and pe is the classification error probability. This assumption makes the probability of misclassification in the positive class smaller than that in the negative class. From the above assumption, we have(33)pY=y¯|Z=y=1+c1c1pe,pY=y|Z=y=c11−y/2−1+c1c1pe.Substituting (33) into (31), we have(34)pY=y|X=x,hx=c11−y/2−1+c1c1pepZ=y|X=x,hx+1+c1c1pepZ=y¯|X=x,hx=c11−y/2−1+c1c1pe1−pZ=y¯|X=x,hx+1+c1c1pepZ=y¯|X=x,hx.Substituting (34) into (26), we have(35)∑i=1nDE+F≤2,where(36)D=Dij−1,E=c11−y/2−1+c1c1pej1−pZ=yi¯|X=xi,hxi,F=1+c1c1pejpZ=yi¯|X=xi.where pej is the error parameter of the jth weak classifier. Simplifying (35) as(37)∑i:htxi=yiDij−1c11−y/2−1+c1/c1pej+∑i:htxi≠yiDij−11+c1/c1pej≤2,we have(38)∑i:htxi=yiDij−11−1+c1/c1pej+∑i:htxi≠yiDij−11+c1/c1pej≤∑i:htxi=yiDij−1c11−y/2−1+c1/c1pej+∑i:htxi≠yiDij−11+c1/c1pej≤2.Setϵj=∑i:htxi≠yiDij−1 and ϵjc=∑i:htxi≠yic11−yi/2Dij−1, and then we have(39)ϵjc≤ϵj≤1+c1c1pej≤12.Parameterpej is defined as(40)pej=c1∑ic11−yi/21+c1e−αj+c2eαj+c2+e−αj+c2,where c2 is positive.Lemma 2.
Whenc1≤pej/ϵjc−pej and c2≤1/2ln∑ic11−yi/2/ϵjc−1, choose the lower boundary Pej=c1/1+c1ϵjc to obtain αj from (40):(41)αj=12ln∑ic11−yi/2−ϵjcϵjc−c2=12ln∑i:htxi=yiDitc11−yi/2∑i:htxi≠yiDitc11−yi/2−c2.In Lemma2, c1≤pej/ϵjc−pej can ensure that the choice of pej satisfies (40) and c2≤1/2ln∑ic11−yi/2/ϵjc−1 is necessary for the positive property of αj.The value ofpY=y¯|X=x,hx can be computed based on (33):(42)pY=y¯|X=x,hx=c11−y/2−1+c1c1pejpZ=y¯|X=x,hx+1+c1c1pejc11−y/2−pZ=y¯|X=x,hx=1+c1c1pej,ifhx=y,c11−y/2−1+c1c1pej,ifhx≠y.Substituting the value ofpej given in (40) into (42), we have(43)pyi¯|xi,hxi=c11−yi/2e−αj+c2eαj+c2+e−αj+c2ifhx=y,c11−yi/2−c11−yi/2e−αj+c2eαj+c2+e−αj+c2ifhx≠y,=c11−yi/2e−yihjxiαj+c2eαj+c2+e−αj+c2.Recalling (29), we have(44)Dij=c11−yi/2e−yihjxiαj+c2Dij−1/Qij∑i=1Nc11−yi/2e−yihjxiαj+c2Dij−1/Qij,where Qij is the normalization term given by(45)Qij=c11−yi/2e−yihjxiαj+c2pyi¯|xi,h1xi,…,hj−1xi+c11−yi/2eyihjxiαj+c2pyi|xi,h1xi,…,hj−1xi.Now, we have given a statistical perspective to see our proposed Algorithm4. However, the statistical approach gives us some deeper information about this algorithm. Lemma 2 gives the conditions of c1≤pej/ϵjc−pej and c2≤1/2ln∑ic11−yi/2/ϵjc−1. The choice of c1 and c2 hasto satisfy these assumptions in our presented algorithm.


## 5. Numerical Studies

### 5.1. Simulation Studies

This section illustrates how to determine the cost parametersc1 and c2 in our model. A simulated dataset is employed to test our proposed model’s performance.


#### 5.1.1. Datasets

Five two-dimensional simulated Gaussian data are chosen, and they are distributed as(46)Xij=Xij1Xij2∼indepN2μi,Σi,where i=−1,1 and j=1,…,ni. We set(47)u−1=78,u1=1315,Σ−1=10338,Σ1=1002.All the variables inX−1,j and X1,j have the class labels −1 (majority) and 1 (minority), respectively. We set five kinds of sizes, np,nn=100,100,50,100,25,100,10,100, and 5,100, to illustrate the performance of different algorithms in different-sized data. b=1,2,4,10, 20 for these five cases. A larger b indicates a more severely imbalanced dataset. Figure 2 shows the training datasets.Figure 2
Simulated Gaussian data for numeric studies.


#### 5.1.2. Parameters Setup

The statistical analysis shows that the cost parametersc1 and c2 have to satisfy(48)c1≤pej1/2−pej≤pejϵjc−pej,(49)c2≤12ln2nc1−1≤12ln2∑ic11−yi/2−1≤12ln∑ic11−yi/2ϵjc−1.We set the upper boundaries as the chosen values ofc1 and c2 in (48) and (49). The restrictions of ϵjc<1/2 and c1≤1 indicate pej≤1/2ϵjc≤1/4. Equation (49) requires that 2nc1−1>0, which indicates c1>1/2n.We choose the values ofpej as 1/4,1/8,1/16,1/32,1/64, and 1/128 to ensure that both cost parameters have the values. Table 4 lists the cost parameters in the present five simulated datasets.Table 4
The cost parameter determinations.
pej1/41/81/161/321/641/128c111/31/71/151/311/63c2b=12.99452.44272.01401.62261.23840.8385b=22.85022.29761.86711.47221.08040.6625b=42.75872.20541.77361.37580.97750.5440b=102.69452.14061.70771.30750.90390.4566b=202.67122.11711.68361.28250.87670.4236


#### 5.1.3. Evaluation Criteria

Twelve criteria listed in Table5 are used to evaluate the performance of algorithms in this paper, including the error rate criteria re and the misclassified cost criteria ce.Table 5
The criteria for classification evaluation.
Training Data Global Error Rateregtrain=yitrain≠yi∗train/nTraining data positive class error ratereptrain=yitrain≠yi∗train&yitrain=1/npTraining data global error costcegtrain=ci⋅yitrain≠yi∗train/∑iciSame size test data global error rateregtest=yitest≠yi∗test/nSame size test data positive class error ratereptest=yitest≠yi∗test&yitest=1/npSame size test data global error costcegtest=ci⋅yitest≠yi∗test/∑iciSmaller size test data global error rateregstest=yistest≠yi∗stest/nsSmaller size test data positive class error raterepstest=yistest≠yi∗stest&yistest=1/npsSmaller size test data global error costcegstest=ci⋅yistest≠yi∗stest/∑iciLarger size test data global error rateregltest=yiltest≠yi∗ltest/nlLarger size test data positive class error raterepltest=yiltest≠yi∗ltest&yiltest=1/nplLarger size test data global error costcegltest=ci⋅yiltest≠yi∗ltest/∑iciThe calculations ofregtest, reptest, and cegtest use the same-sized test data as the training data, ntrain=ntest=n,nptrain=nptest=np. Small-sized and large-sized test data are used for the calculations of regstest,repstest, and cegstest and regltest,repltest, and cegltest, respectively. It means ns<n<nl,nps<np<npl. Note that the positive class error cost is equal to the error rate since the positive cost parameter is set as 1. All the test datasets in this paper keep the same imbalance index b as the training data, namely,(50)S−testS+test=S−trainS+train=b.


#### 5.1.4. Results of Simulation Studies

We run every simulated dataset 100 times and calculate the mean and standard deviation values as the results. Tables6–10 list the experiments. Obviously, with the imbalanced index b growing larger, the smaller cost parameters could better classify performances. We conjecture that the classification performance is best when the corresponding c1 is closest to 1/b.Table 6
The experiment results ofb=1.
c1=1c2=2.9945c1=1/3c2=2.4427c1=1/7c2=2.0140c1=1/15c2=1.6226c1=1/31c2=1.2384c1=1/63c2=0.8385regtrain0.0030(0.0438)0.0036 (0.0393)0.0035 (0.0310)0.0039 (0.0334)0.0040 (0.0554)0.0042 (0.0612)reptrain0.0031(0.0448)0.0034 (0.0329)0.0036 (0.0426)0.0039 (0.0381)0.0040 (0.0796)0.0041 (0.0540)cegtrain0.0030(0.0318)0.0034 (0.0538)0.0032 (0.0685)0.0035 (0.0317)0.0036 (0.0696)0.0038 (0.0745)regtest0.0031(0.0339)0.0031(0.0376)0.0032 (0.0657)0.0037 (0.0331)0.0039 (0.0300)0.0040 (0.0496)reptest0.0031 (0.0323)0.0036 (0.0537)0.0030(0.0849)0.0038 (0.0415)0.0039 (0.0338)0.0041 (0.0900)cegtest0.0031 (0.0408)0.0030(0.0644)0.0031 (0.0593)0.0035 (0.0595)0.0038 (0.0384)0.0039 (0.0563)regstest0.0031(0.0477)0.0033 (0.0333)0.0032 (0.0640)0.0037 (0.0446)0.0039 (0.0755)0.0042 (0.0620)repstest0.0031 (0.0332)0.0033 (0.0346)0.0030(0.0748)0.0039 (0.0536)0.0041 (0.0709)0.0042 (0.0657)cegstest0.0030(0.0439)0.0032 (0.0572)0.0031 (0.0554)0.0035 (0.0390)0.0036 (0.0420)0.0040 (0.0565)regltest0.0029(0.0323)0.0035 (0.0780)0.0031 (0.0437)0.0039 (0.0667)0.0040 (0.0434)0.00041 (0.0887)repltest0.0030(0.0478)0.0031 (0.0365)0.0032 (0.0776)0.0038 (0.0662)0.0040 (0.0835)0.0043 (0.0499)cegltest0.0029(0.0351)0.0034 (0.0522)0.0030 (0.0676)0.0037 (0.0334)0.0038 (0.0341)0.0040 (0.0344)Table 7
The experiment results ofb=2.
c1=1c2=2.9945c1=1/3c2=2.4427c1=1/7c2=2.0140c1=1/15c2=1.6226c1=1/31c2=1.2384c1=1/63c2=0.8385regtrain0.0043 (0.4673)0.0020(0.4204)0.0027 (0.4370)0.0026 (0.4246)0.0029 (0.3300)0.0033 (0.3555)reptrain0.0053 (0.3950)0.0026 (0.4368)0.0025(0.4637)0.0026 (0.4645)0.0026 (0.3891)0.0029 (0.4967)cegtrain0.0043 (0.4082)0.0022(0.3104)0.0024 (0.4686)0.0024 (0.4837)0.0027 (0.4853)0.0030 (0.3063)regtest0.0038 (0.3738)0.0021(0.4547)0.0035 (0.4430)0.0027 (0.3696)0.0029 (0.3119)0.0029 (0.4192)reptest0.0045 (0.3145)0.0028 (0.4728)0.0025(0.4966)0.0028 (0.3499)0.0031 (0.4755)0.0033 (0.4287)cegtest0.0038 (0.4071)0.0020(0.4748)0.0029 (0.4787)0.0027 (0.3140)0.0029 (0.4898)0.0029 (0.4157)regstest0.0042 (0.3660)0.0016(0.3660)0.0024 (0.3477)0.0029 (0.4636)0.0029 (0.4353)0.0031 (0.3849)repstest0.0055 (0.4836)0.0018(0.3095)0.0032 (0.3323)0.0032 (0.3091)0.0026 (0.4430)0.0031 (0.4331)cegstest0.0042 (0.3592)0.0025 (0.4214)0.0022(0.4206)0.0026 (0.4411)0.0028 (0.3444)0.0028 (0.3741)regltest0.0042 (0.3248)0.0018(0.3845)0.0026 (0.4646)0.0024 (0.4188)0.0028 (0.4434)0.0033 (0.3432)repltest0.0054 (0.4604)0.0020(0.3554)0.0026 (0.3905)0.0025 (0.3730)0.0031 (0.3185)0.0030 (0.3504)cegltest0.0042 (0.4394)0.0018(0.3538)0.0025 (0.3312)0.0023 (0.3973)0.0027 (0.3230)0.0031 (0.4799)Table 8
The experiment results ofb=5.
c1=1c2=2.9945c1=1/3c2=2.4427c1=1/7c2=2.0140c1=1/15c2=1.6226c1=1/31c2=1.2384c1=1/63c2=0.8385regtrain0.0062 (0.4142)0.0032 (0.3957)0.0025 (0.3406)0.0031 (0.2384)0.0032 (0.2024)0.0032 (0.2019)reptrain0.0089 (0.3637)0.0050 (0.4967)0.0030 (0.4904)0.0031 (0.3888)0.0031 (0.4530)0.0033 (0.2667)cegtrain0.0062 (0.2088)0.0030 (0.2824)0.0024 (0.4008)0.0030 (0.3674)0.0031 (0.2040)0.0031 (0.3920)regtest0.0061 (0.2904)0.0028 (0.2729)0.0026 (0.4036)0.0030 (0.4861)0.0030 (0.4195)0.0033 (0.4460)reptest0.0090 (0.4453)0.0058 (0.3510)0.0027 (0.3723)0.0033 (0.3545)0.0032 (0.2935)0.0033 (0.2351)cegtest0.0061 (0.3162)0.0027 (0.2519)0.0024 (0.4383)0.0029 (0.3922)0.0028 (0.3349)0.0031 (0.2821)regstest0.0063 (0.4622)0.0028 (0.4731)0.0029 (0.4915)0.0033 (0.4227)0.0031 (0.3523)0.0031 (0.3897)repstest0.0093 (0.4590)0.0051 (0.4006)0.0025 (0.3267)0.0033 (0.3091)0.0032 (0.3934)0.0033 (0.2366)cegstest0.0063 (0.4888)0.0027 (0.3809)0.0027 (0.3463)0.0030 (0.2130)0.0030 (0.4935)0.0028 (0.4646)regltest0.0068 (0.3685)0.0032 (0.2169)0.0026 (0.2913)0.0030 (0.4536)0.0032 (0.4427)0.0031 (0.3334)repltest0.0088 (0.4268)0.0050 (0.2679)0.0030 (0.3359)0.0033 (0.2583)0.0031 (0.3136)0.0031 (0.2392)cegltest0.0068 (0.2535)0.0029 (0.3750)0.0024 (0.2735)0.0028 (0.3600)0.0030 (0.3842)0.0030 (0.4166)Table 9
The experiment results ofb=10.
c1=1c2=2.9945c1=1/3c2=2.4427c1=1/7c2=2.0140c1=1/15c2=1.6226c1=1/31c2=1.2384c1=1/63c2=0.8385regtrain0.2131 (0.3640)0.0629 (0.3041)0.0029 (0.5293)0.0028(0.3091)0.0029 (0.3091)0.0029 (0.3396)reptrain0.4229 (0.5814)0.2629 (0.4365)0.0029 (0.4660)0.0028(0.5930)0.0028(0.5930)0.0031 (0.5406)cegtrain0.2030 (0.4871)0.0630 (0.5596)0.0028(0.5697)0.0030 (0.3245)0.0030 (0.3245)0.0030 (0.5110)regtest0.2533 (0.2023)0.0630 (0.4073)0.0030 (0.3995)0.0028 (0.2032)0.0028 (0.2032)0.0027(0.4745)reptest0.4531 (0.5172)0.2630 (0.3888)0.0028(0.3106)0.0030 (0.4138)0.0030 (0.4138)0.0028(0.4054)cegtest0.2629 (0.5811)0.0631 (0.4503)0.0029 (0.5217)0.0028(0.4158)0.0028(0.4158)0.0033 (0.2670)regstest0.2530 (0.4356)0.0629 (0.4585)0.0030(0.3506)0.0030(0.2349)0.0030(0.2349)0.0033 (0.4934)repstest0.4230 (0.3356)0.2629 (0.2448)0.0029(0.5382)0.0029(0.3647)0.0030 (0.3647)0.0033 (0.5974)cegstest0.2231 (0.3216)0.0630 (0.3403)0.0030 (0.5239)0.0028(0.3297)0.0028(0.3297)0.0033 (0.4633)regltest0.3029 (0.3406)0.0629 (0.4907)0.0028(0.3624)0.0029 (0.5111)0.0029 (0.5111)0.0030 (0.3396)repltest0.4031 (0.2818)0.2629 (0.2790)0.0029 (0.2387)0.0028(0.5565)0.0029 (0.5565)0.0030 (0.3725)cegltest0.3030 (0.5284)0.0629 (0.5398)0.0029 (0.3888)0.0027(0.4037)0.0027(0.4037)0.0028 (0.5488)Table 10
The experiment results ofb=20.
c1=1c2=2.9945c1=1/3c2=2.4427c1=1/7c2=2.0140c1=1/15c2=1.6226c1=1/31c2=1.2384c1=1/63c2=0.8385regtrain0.2032 (0.2626)0.1233 (0.3151)0.0829 (0.4914)0.0030 (0.5119)0.0029 (0.4911)0.0033 (0.5357)reptrain0.5730 (0.2680)0.3132 (0.4644)0.1330 (0.3304)0.0026 (0.5655)0.0026 (0.2402)0.0029 (0.3079)cegtrain0.2033 (0.2633)0.1232 (0.4608)0.0833 (0.4054)0.0030 (0.2351)0.0027 (0.2147)0.0030 (0.5034)regtest0.2236 (0.5247)0.1729 (0.3532)0.0817 (0.3587)0.0027 (0.5364)0.0029 (0.5309)0.0029 (0.4907)reptest0.5432 (0.3757)0.3730 (0.5249)0.1360 (0.2709)0.0028 (0.4515)0.0031 (0.2166)0.0033 (0.4749)cegtest0.2733 (0.2104)0.1731 (0.3326)0.0829 (0.3310)0.0027 (0.4997)0.0029 (0.3874)0.0029 (0.4969)regstest0.2830 (0.3424)0.1632 (0.4303)0.0932 (0.5882)0.0029 (0.4927)0.0029 (0.4914)0.0031 (0.5735)repstest0.5729 (0.5996)0.3829 (0.5110)0.2532 (0.5348)0.0032 (0.4090)0.0026 (0.4582)0.0031 (0.3844)cegstest0.2931 (0.2220)0.1531 (0.2105)0.0866 (0.3779)0.0026 (0.2612)0.0030 (0.4546)0.0028 (0.5995)regltest0.2429 (0.2612)0.1733 (0.2488)0.0832 (0.5526)0.0024 (0.2521)0.0028 (0.4023)0.0033 (0.2909)repltest0.6031 (0.2359)0.4029 (0.3856)0.2772 (0.3256)0.0025 (0.5423)0.0031 (0.5677)0.0030 (0.2734)cegltest0.2329 (0.2745)0.1631 (0.4322)0.0830 (0.3358)0.0031 (0.4016)0.0027 (0.2118)0.0031 (0.4080)


### 5.2. Real Data Studies

Four real binary datasets are studied, and the values of their imbalanced indexb are 15.13, 11.59, 5.14, and 3.36. Since all the datasets are imbalanced, each dataset is randomly split in half as the training part and half as the test part to avoid the situation that there are no or too few positive samples in the training or test data. We choose thirteen popular classification algorithms in Table 11 for comparisons. The first five are all the traditional ones and the sixth, PRVM [48], is designed for the imbalanced data problem, which has achieved convincing results. The last seven ones are the traditional cost-sensitive classifiers, which we have discussed in the discussion part. Note that the AdaImC algorithm in this part is tested based on the different determinations of the cost factors to choose the optimal parameter setup before joining the comparison. We will not show this process again in the real data studies.Table 11
Compared machine learning algorithms in real data studies.
NameDescriptionParametersSVMsStandard single support vector machines with RBF kernelγbest is obtained by grid searching in 2−20:20Ada-DTAdaBoost using decision tree as weak classifiersAda-LSVMsAdaBoost using linear kernel SVMs as weak classifiersAda-RSVMsAdaBoost using RBF kernel SVMs as weak classifiersγbest is obtained by grid searching in 2−20:20RVMStandard single relevance vector machines with RBF kernelγbest is obtained by grid searching in 2−20:20PRVMRelevance vector machines under the probit link function with RBF kernelγbest is obtained by grid searching in 2−20:20ICETIncorporates misclassification costs in the fitness function of genetic algorithmsCost-sensitive decision treesUse the misclassification costs in tree building processMetaCostConvert cost-insensitive classifiers into cost-sensitive ones without modifyingCost-sensitive naive BayesUse cost factor information as the priorEmpirical thresholdingUse bagging on decision trees to obtain reliable probability estimationsCostingUse rejection sampling to resample the dataWeightingAssign a normalized weight to each instance according to the misclassification costs


#### 5.2.1. Seismic-Bumps Data Study

The seismic-bumps data withb=15.13 are obtained from the UCI Machine Learning Repository [49]. This dataset is about the seismic hazard in mining activity. Mining activity is always connected with the occurrence of dangers, which are called mining hazards. Seismic hazard is a special and dangerous case in mining hazards. The seismic-bumps dataset is intensely imbalanced with only 170 samples of the positive class among a total of 2584 samples, and it contains 18 explanatory variables, including seismic hazard assessment obtained by different methods, seismic energy recorded by different types of equipment, and the number of seismic bumps in different energy range. The hazardous state is the negative class defined as a high energy seismic bump occurring in the next shift, which is indicated by y=−1. The nonhazardous state is the positive class defined as no high energy seismic bump occurring in the next shift, which is indicated by y=1.Table12 displays the seismic-bumps data experiment. It shows the great performance of our proposed algorithms for the classification of this dataset. Except for the specificity, all evaluation criteria of the PRVM and the AdaImC perform better than other algorithms. The AdaImC obviously outperforms the PRVM except for the F−1Measure.Table 12
The results of seismic-bumps datab=15.13.
AlgorithmsEvaluation measuresSensitivitySpecificityF-1 MeasureAUCGlobal AccuracySVMs0.00001.00000.00000.50.9320Ada-DT0.10470.98730.16360.54600.9273Ada-LSVMs0.00000.99920.00000.49960.9312Ada-RSVMs0.62790.66750.20300.64770.6648RVM0.53180.97550.86010.21010.6533PRMV0.90110.93640.96010.91010.9473ICET0.91130.95780.92210.91220.9361Cost-sensitive decision trees0.88040.93820.91300.91430.9473MetaCost0.90430.94610.93160.90060.9422Cost-sensitive naive Bayes0.90180.94180.90880.92000.9461Empirical thresholding0.89330.96550.92870.91550.9327Costing0.91610.92850.91290.92670.9378Weighting0.90650.94990.90160.93310.9502AdaImC0.92770.98530.91120.93760.9588


#### 5.2.2. Glass-2 Data Study

The Glass-2 dataset is obtained from the Knowledge Extraction based on Evolutionary Learning (KEEL) Dataset Repository [50]. This dataset is originally from the US Forensic Science Service and intensely imbalanced with only 17 samples of the positive class among a total of 214 samples, which means b=11.59. The Glass-2 dataset has eight explanatory variables, including the refractive index and the different chemical elements content. The positive class is labeled as the glass made by the nonfloat process and indicated by y=1. The negative class is defined as the glass made by other processed methods and indicated by y=−1.Table13 displays the Glass-2 data experiment. It shows similar results as the seismic-bumps data. Except for the Specificity, all evaluation measures of PRVM and AdaImC are better than other algorithms. Compared to the PRVM, our proposed AdaImC has a slight improvement.Table 13
The results of Glass-2 datab=11.59.
AlgorithmsEvaluation measuresSensitivitySpecificityF-1 MeasureAUCGlobal AccuracySVMs0.00001.00000.00000.50.8972Ada-DT0.09090.98960.15380.54020.8972Ada-LSVMs0.00001.00000.00000.50000.8970Ada-RSVMs0.18180.90630.18180.54400.8318RVM0.14330.95880.17100.60310.8313PRMV0.92670.90880.94330.92760.9006ICET0.91330.95260.93760.92730.9361Cost-sensitive decision trees0.90630.97270.92880.91880.9422MetaCost0.91220.95280.94760.93810.9418Cost-sensitive naive Bayes0.90050.93170.97020.94270.9355Empirical thresholding0.89730.94660.94660.93290.9341Costing0.91210.97520.93100.94040.9362Weighting0.90330.94990.93410.93870.9132AdaImC0.92930.90190.95180.97860.9470


#### 5.2.3. New-Thyroid-1 Data Study

The new-thyroid-1 dataset is also obtained from the KEEL Dataset Repository. This dataset is originally from the Garvan Institute in Sydney, Australia. It includes the information of thyroid patients and is rearranged by KEEL to be an imbalanced binary dataset. The new-thyroid-1 dataset is imbalanced with 35 positive samples among a total of 215 samples, which indicates the imbalanced index isb=5.14. This dataset contains 5 explanatory variables, including the levels of different hormones such as thyroxin and triiodothyronine. The positive class is labeled as hyperthyroidism, which is indicated by y=1. The negative class is defined as nonhyperthyroidism, which is indicated by y=−1.Table14 displays the results of the new-thyroid-1 data experiment. It shows the satisfactory performance of our proposed algorithm compared to others for this data classification problem. Almost all the evaluation criteria of the PRVM and the AdaImC perform better than other algorithms. Because b=5.14 is a relatively small number in this dataset, the differences between PRVM and AdaImC are not significant.Table 14
The results of new-thyroid-1 datab=5.14.
AlgorithmsEvaluation measuresSensitivitySpecificityF-1 MeasureAUCGlobal AccuracySVMs0.26321.00000.41670.63160.8704Ada-DT0.78950.98880.85710.88910.9537Ada-LSVMs0.84210.98880.88890.91540.9630Ada-RSVMs0.84210.96630.84210.90420.9444RVM0.83110.94720.87320.90310.9135PRMV0.91330.98460.95720.97310.9655ICET0.91720.96790.92210.91220.9361Cost-sensitive decision trees0.89540.97550.91300.91430.9473MetaCost0.90070.94210.93160.90060.9422Cost-sensitive naive Bayes0.91100.90490.90880.92000.9461Empirical thresholding0.88940.94820.92870.91550.9327Costing0.91610.95620.93310.92670.9378Weighting0.90480.95730.90160.93310.9502AdaImC0.91480.98330.96200.97440.9670


#### 5.2.4. Ecoli-1 Data Study

The Ecoli-1 dataset is also obtained from the KEEL Dataset Repository. This dataset initially comes from the Institute of Molecular and Cellular Biology at Osaka University. It includes the information of the cellular localization sites of proteins and is rearranged by KEEL as an imbalanced binary dataset. The Ecoli-1 dataset is imbalanced with 77 positive samples among a total of 336 samples, which indicates the imbalanced index isb=3.36. This dataset contains seven explanatory variables, including the signal sequence recognition scores by different methods and the presence of charge on N-terminus. The positive class is labeled as the inner membrane without a signal sequence, which is indicated by y=1. The negative class is defined as other situations, which is indicated by y=−1.Table15 displays the results of the Ecoli-1 data experiment. It shows all the algorithms perform similarly since the imbalanced situation is not obvious. Our proposed AdaImC still has a convincing performance for this balanced dataset.Table 15
The results of Ecoli-1 datab=3.36.
AlgorithmsEvaluation measuresSensitivitySpecificityF-1 MeasureAUCGlobal AccuracySVMs0.88890.92420.82050.90660.9167Ada-DT0.86110.90910.78480.88510.8988Ada-LSVMs0.00001.00000.00000.50000.7857Ada-RSVMs0.94440.86360.77270.90400.8810RVM0.95310.91430.83310.92120.9145PRMV0.97790.94310.98220.92210.9055ICET0.94660.95880.96420.90670.8730Cost-sensitive decision trees0.95120.98310.95800.91220.8933MetaCost0.94880.97930.97310.90520.8517Cost-sensitive naive Bayes0.95710.96830.97380.91800.7841Empirical thresholding0.93700.96810.96390.90330.8905Costing0.95190.94880.94880.90730.9022Weighting0.96110.97690.95810.91800.8467AdaImC0.96440.93550.98830.92750.9108Our proposed AdaImC algorithm outperforms the traditional ones in imbalanced data classification based on real data studies. With a large imbalanced indexb, our proposed algorithm shows excellent performance.


### 5.3. Algorithm Complexity Analysis

Our proposed AdaImC algorithm outperforms the classical AdaBoost, especially in the imbalanced data problem. However, it has a more complex structure, which means the model complexity is larger than the classic one. In practice, especially for big data, our proposed one may face an inefficiency issue.Assume thatc1 has been determined. Compared with the classic AdaBoost (Algorithm 1), the AdaImC (Algorithm 4) has the following additional computation for the training dataset S:Step 0: “n+2n,” where“n” is the calculated amount for Dti=c11−yi/2 and “2n” is the calculated amount for the(51)0<c2<12ln∑i:htxi=yiDitc11−yi/2∑i:htxi≠yiDitc11−yi/2.Step 2: “2n” is the calculated amount for(52)αt=12ln∑i:htxi=yiDitc11−yi/2∑i:htxi≠yiDitc11−yi/2−c2.Step 3: “3n” is the calculated amount for(53)Dt+1i=c11−yi/2Dtiexp−yihtxiαt+c2n∏t=1TZt.Totally, AdaImC has the additional8n computations than the classic AdaBoost, which seems acceptable for the practice. We compare the code running time for AdaImC and the classic AdaBoost based on the above four real datasets in Table 16. In relatively big datasets, the AdaImC needs significantly more time in practice. The additional time for code running is acceptable, and the complexity does not lower the usability of our proposed algorithm based on this limited study.Table 16
The results of algorithms’ runningTimea.
AlgorithmsCode running time (seconds)Seismic-bumpsGlass-2New-thyroid-1Ecoli-1AdaBoost54.3336.1229.8027.00AdaImC61.7340.5131.2232.91Increase percentage13.62%12.15%4.77%21.89%aPerformed in Python 3.6 on macOS 10.12.6.


## 5.1. Simulation Studies

This section illustrates how to determine the cost parametersc1 and c2 in our model. A simulated dataset is employed to test our proposed model’s performance.


### 5.1.1. Datasets

Five two-dimensional simulated Gaussian data are chosen, and they are distributed as(46)Xij=Xij1Xij2∼indepN2μi,Σi,where i=−1,1 and j=1,…,ni. We set(47)u−1=78,u1=1315,Σ−1=10338,Σ1=1002.All the variables inX−1,j and X1,j have the class labels −1 (majority) and 1 (minority), respectively. We set five kinds of sizes, np,nn=100,100,50,100,25,100,10,100, and 5,100, to illustrate the performance of different algorithms in different-sized data. b=1,2,4,10, 20 for these five cases. A larger b indicates a more severely imbalanced dataset. Figure 2 shows the training datasets.Figure 2
Simulated Gaussian data for numeric studies.


### 5.1.2. Parameters Setup

The statistical analysis shows that the cost parametersc1 and c2 have to satisfy(48)c1≤pej1/2−pej≤pejϵjc−pej,(49)c2≤12ln2nc1−1≤12ln2∑ic11−yi/2−1≤12ln∑ic11−yi/2ϵjc−1.We set the upper boundaries as the chosen values ofc1 and c2 in (48) and (49). The restrictions of ϵjc<1/2 and c1≤1 indicate pej≤1/2ϵjc≤1/4. Equation (49) requires that 2nc1−1>0, which indicates c1>1/2n.We choose the values ofpej as 1/4,1/8,1/16,1/32,1/64, and 1/128 to ensure that both cost parameters have the values. Table 4 lists the cost parameters in the present five simulated datasets.Table 4
The cost parameter determinations.
pej1/41/81/161/321/641/128c111/31/71/151/311/63c2b=12.99452.44272.01401.62261.23840.8385b=22.85022.29761.86711.47221.08040.6625b=42.75872.20541.77361.37580.97750.5440b=102.69452.14061.70771.30750.90390.4566b=202.67122.11711.68361.28250.87670.4236


### 5.1.3. Evaluation Criteria

Twelve criteria listed in Table5 are used to evaluate the performance of algorithms in this paper, including the error rate criteria re and the misclassified cost criteria ce.Table 5
The criteria for classification evaluation.
Training Data Global Error Rateregtrain=yitrain≠yi∗train/nTraining data positive class error ratereptrain=yitrain≠yi∗train&yitrain=1/npTraining data global error costcegtrain=ci⋅yitrain≠yi∗train/∑iciSame size test data global error rateregtest=yitest≠yi∗test/nSame size test data positive class error ratereptest=yitest≠yi∗test&yitest=1/npSame size test data global error costcegtest=ci⋅yitest≠yi∗test/∑iciSmaller size test data global error rateregstest=yistest≠yi∗stest/nsSmaller size test data positive class error raterepstest=yistest≠yi∗stest&yistest=1/npsSmaller size test data global error costcegstest=ci⋅yistest≠yi∗stest/∑iciLarger size test data global error rateregltest=yiltest≠yi∗ltest/nlLarger size test data positive class error raterepltest=yiltest≠yi∗ltest&yiltest=1/nplLarger size test data global error costcegltest=ci⋅yiltest≠yi∗ltest/∑iciThe calculations ofregtest, reptest, and cegtest use the same-sized test data as the training data, ntrain=ntest=n,nptrain=nptest=np. Small-sized and large-sized test data are used for the calculations of regstest,repstest, and cegstest and regltest,repltest, and cegltest, respectively. It means ns<n<nl,nps<np<npl. Note that the positive class error cost is equal to the error rate since the positive cost parameter is set as 1. All the test datasets in this paper keep the same imbalance index b as the training data, namely,(50)S−testS+test=S−trainS+train=b.


### 5.1.4. Results of Simulation Studies

We run every simulated dataset 100 times and calculate the mean and standard deviation values as the results. Tables6–10 list the experiments. Obviously, with the imbalanced index b growing larger, the smaller cost parameters could better classify performances. We conjecture that the classification performance is best when the corresponding c1 is closest to 1/b.Table 6
The experiment results ofb=1.
c1=1c2=2.9945c1=1/3c2=2.4427c1=1/7c2=2.0140c1=1/15c2=1.6226c1=1/31c2=1.2384c1=1/63c2=0.8385regtrain0.0030(0.0438)0.0036 (0.0393)0.0035 (0.0310)0.0039 (0.0334)0.0040 (0.0554)0.0042 (0.0612)reptrain0.0031(0.0448)0.0034 (0.0329)0.0036 (0.0426)0.0039 (0.0381)0.0040 (0.0796)0.0041 (0.0540)cegtrain0.0030(0.0318)0.0034 (0.0538)0.0032 (0.0685)0.0035 (0.0317)0.0036 (0.0696)0.0038 (0.0745)regtest0.0031(0.0339)0.0031(0.0376)0.0032 (0.0657)0.0037 (0.0331)0.0039 (0.0300)0.0040 (0.0496)reptest0.0031 (0.0323)0.0036 (0.0537)0.0030(0.0849)0.0038 (0.0415)0.0039 (0.0338)0.0041 (0.0900)cegtest0.0031 (0.0408)0.0030(0.0644)0.0031 (0.0593)0.0035 (0.0595)0.0038 (0.0384)0.0039 (0.0563)regstest0.0031(0.0477)0.0033 (0.0333)0.0032 (0.0640)0.0037 (0.0446)0.0039 (0.0755)0.0042 (0.0620)repstest0.0031 (0.0332)0.0033 (0.0346)0.0030(0.0748)0.0039 (0.0536)0.0041 (0.0709)0.0042 (0.0657)cegstest0.0030(0.0439)0.0032 (0.0572)0.0031 (0.0554)0.0035 (0.0390)0.0036 (0.0420)0.0040 (0.0565)regltest0.0029(0.0323)0.0035 (0.0780)0.0031 (0.0437)0.0039 (0.0667)0.0040 (0.0434)0.00041 (0.0887)repltest0.0030(0.0478)0.0031 (0.0365)0.0032 (0.0776)0.0038 (0.0662)0.0040 (0.0835)0.0043 (0.0499)cegltest0.0029(0.0351)0.0034 (0.0522)0.0030 (0.0676)0.0037 (0.0334)0.0038 (0.0341)0.0040 (0.0344)Table 7
The experiment results ofb=2.
c1=1c2=2.9945c1=1/3c2=2.4427c1=1/7c2=2.0140c1=1/15c2=1.6226c1=1/31c2=1.2384c1=1/63c2=0.8385regtrain0.0043 (0.4673)0.0020(0.4204)0.0027 (0.4370)0.0026 (0.4246)0.0029 (0.3300)0.0033 (0.3555)reptrain0.0053 (0.3950)0.0026 (0.4368)0.0025(0.4637)0.0026 (0.4645)0.0026 (0.3891)0.0029 (0.4967)cegtrain0.0043 (0.4082)0.0022(0.3104)0.0024 (0.4686)0.0024 (0.4837)0.0027 (0.4853)0.0030 (0.3063)regtest0.0038 (0.3738)0.0021(0.4547)0.0035 (0.4430)0.0027 (0.3696)0.0029 (0.3119)0.0029 (0.4192)reptest0.0045 (0.3145)0.0028 (0.4728)0.0025(0.4966)0.0028 (0.3499)0.0031 (0.4755)0.0033 (0.4287)cegtest0.0038 (0.4071)0.0020(0.4748)0.0029 (0.4787)0.0027 (0.3140)0.0029 (0.4898)0.0029 (0.4157)regstest0.0042 (0.3660)0.0016(0.3660)0.0024 (0.3477)0.0029 (0.4636)0.0029 (0.4353)0.0031 (0.3849)repstest0.0055 (0.4836)0.0018(0.3095)0.0032 (0.3323)0.0032 (0.3091)0.0026 (0.4430)0.0031 (0.4331)cegstest0.0042 (0.3592)0.0025 (0.4214)0.0022(0.4206)0.0026 (0.4411)0.0028 (0.3444)0.0028 (0.3741)regltest0.0042 (0.3248)0.0018(0.3845)0.0026 (0.4646)0.0024 (0.4188)0.0028 (0.4434)0.0033 (0.3432)repltest0.0054 (0.4604)0.0020(0.3554)0.0026 (0.3905)0.0025 (0.3730)0.0031 (0.3185)0.0030 (0.3504)cegltest0.0042 (0.4394)0.0018(0.3538)0.0025 (0.3312)0.0023 (0.3973)0.0027 (0.3230)0.0031 (0.4799)Table 8
The experiment results ofb=5.
c1=1c2=2.9945c1=1/3c2=2.4427c1=1/7c2=2.0140c1=1/15c2=1.6226c1=1/31c2=1.2384c1=1/63c2=0.8385regtrain0.0062 (0.4142)0.0032 (0.3957)0.0025 (0.3406)0.0031 (0.2384)0.0032 (0.2024)0.0032 (0.2019)reptrain0.0089 (0.3637)0.0050 (0.4967)0.0030 (0.4904)0.0031 (0.3888)0.0031 (0.4530)0.0033 (0.2667)cegtrain0.0062 (0.2088)0.0030 (0.2824)0.0024 (0.4008)0.0030 (0.3674)0.0031 (0.2040)0.0031 (0.3920)regtest0.0061 (0.2904)0.0028 (0.2729)0.0026 (0.4036)0.0030 (0.4861)0.0030 (0.4195)0.0033 (0.4460)reptest0.0090 (0.4453)0.0058 (0.3510)0.0027 (0.3723)0.0033 (0.3545)0.0032 (0.2935)0.0033 (0.2351)cegtest0.0061 (0.3162)0.0027 (0.2519)0.0024 (0.4383)0.0029 (0.3922)0.0028 (0.3349)0.0031 (0.2821)regstest0.0063 (0.4622)0.0028 (0.4731)0.0029 (0.4915)0.0033 (0.4227)0.0031 (0.3523)0.0031 (0.3897)repstest0.0093 (0.4590)0.0051 (0.4006)0.0025 (0.3267)0.0033 (0.3091)0.0032 (0.3934)0.0033 (0.2366)cegstest0.0063 (0.4888)0.0027 (0.3809)0.0027 (0.3463)0.0030 (0.2130)0.0030 (0.4935)0.0028 (0.4646)regltest0.0068 (0.3685)0.0032 (0.2169)0.0026 (0.2913)0.0030 (0.4536)0.0032 (0.4427)0.0031 (0.3334)repltest0.0088 (0.4268)0.0050 (0.2679)0.0030 (0.3359)0.0033 (0.2583)0.0031 (0.3136)0.0031 (0.2392)cegltest0.0068 (0.2535)0.0029 (0.3750)0.0024 (0.2735)0.0028 (0.3600)0.0030 (0.3842)0.0030 (0.4166)Table 9
The experiment results ofb=10.
c1=1c2=2.9945c1=1/3c2=2.4427c1=1/7c2=2.0140c1=1/15c2=1.6226c1=1/31c2=1.2384c1=1/63c2=0.8385regtrain0.2131 (0.3640)0.0629 (0.3041)0.0029 (0.5293)0.0028(0.3091)0.0029 (0.3091)0.0029 (0.3396)reptrain0.4229 (0.5814)0.2629 (0.4365)0.0029 (0.4660)0.0028(0.5930)0.0028(0.5930)0.0031 (0.5406)cegtrain0.2030 (0.4871)0.0630 (0.5596)0.0028(0.5697)0.0030 (0.3245)0.0030 (0.3245)0.0030 (0.5110)regtest0.2533 (0.2023)0.0630 (0.4073)0.0030 (0.3995)0.0028 (0.2032)0.0028 (0.2032)0.0027(0.4745)reptest0.4531 (0.5172)0.2630 (0.3888)0.0028(0.3106)0.0030 (0.4138)0.0030 (0.4138)0.0028(0.4054)cegtest0.2629 (0.5811)0.0631 (0.4503)0.0029 (0.5217)0.0028(0.4158)0.0028(0.4158)0.0033 (0.2670)regstest0.2530 (0.4356)0.0629 (0.4585)0.0030(0.3506)0.0030(0.2349)0.0030(0.2349)0.0033 (0.4934)repstest0.4230 (0.3356)0.2629 (0.2448)0.0029(0.5382)0.0029(0.3647)0.0030 (0.3647)0.0033 (0.5974)cegstest0.2231 (0.3216)0.0630 (0.3403)0.0030 (0.5239)0.0028(0.3297)0.0028(0.3297)0.0033 (0.4633)regltest0.3029 (0.3406)0.0629 (0.4907)0.0028(0.3624)0.0029 (0.5111)0.0029 (0.5111)0.0030 (0.3396)repltest0.4031 (0.2818)0.2629 (0.2790)0.0029 (0.2387)0.0028(0.5565)0.0029 (0.5565)0.0030 (0.3725)cegltest0.3030 (0.5284)0.0629 (0.5398)0.0029 (0.3888)0.0027(0.4037)0.0027(0.4037)0.0028 (0.5488)Table 10
The experiment results ofb=20.
c1=1c2=2.9945c1=1/3c2=2.4427c1=1/7c2=2.0140c1=1/15c2=1.6226c1=1/31c2=1.2384c1=1/63c2=0.8385regtrain0.2032 (0.2626)0.1233 (0.3151)0.0829 (0.4914)0.0030 (0.5119)0.0029 (0.4911)0.0033 (0.5357)reptrain0.5730 (0.2680)0.3132 (0.4644)0.1330 (0.3304)0.0026 (0.5655)0.0026 (0.2402)0.0029 (0.3079)cegtrain0.2033 (0.2633)0.1232 (0.4608)0.0833 (0.4054)0.0030 (0.2351)0.0027 (0.2147)0.0030 (0.5034)regtest0.2236 (0.5247)0.1729 (0.3532)0.0817 (0.3587)0.0027 (0.5364)0.0029 (0.5309)0.0029 (0.4907)reptest0.5432 (0.3757)0.3730 (0.5249)0.1360 (0.2709)0.0028 (0.4515)0.0031 (0.2166)0.0033 (0.4749)cegtest0.2733 (0.2104)0.1731 (0.3326)0.0829 (0.3310)0.0027 (0.4997)0.0029 (0.3874)0.0029 (0.4969)regstest0.2830 (0.3424)0.1632 (0.4303)0.0932 (0.5882)0.0029 (0.4927)0.0029 (0.4914)0.0031 (0.5735)repstest0.5729 (0.5996)0.3829 (0.5110)0.2532 (0.5348)0.0032 (0.4090)0.0026 (0.4582)0.0031 (0.3844)cegstest0.2931 (0.2220)0.1531 (0.2105)0.0866 (0.3779)0.0026 (0.2612)0.0030 (0.4546)0.0028 (0.5995)regltest0.2429 (0.2612)0.1733 (0.2488)0.0832 (0.5526)0.0024 (0.2521)0.0028 (0.4023)0.0033 (0.2909)repltest0.6031 (0.2359)0.4029 (0.3856)0.2772 (0.3256)0.0025 (0.5423)0.0031 (0.5677)0.0030 (0.2734)cegltest0.2329 (0.2745)0.1631 (0.4322)0.0830 (0.3358)0.0031 (0.4016)0.0027 (0.2118)0.0031 (0.4080)


## 5.1.1. Datasets

Five two-dimensional simulated Gaussian data are chosen, and they are distributed as(46)Xij=Xij1Xij2∼indepN2μi,Σi,where i=−1,1 and j=1,…,ni. We set(47)u−1=78,u1=1315,Σ−1=10338,Σ1=1002.All the variables inX−1,j and X1,j have the class labels −1 (majority) and 1 (minority), respectively. We set five kinds of sizes, np,nn=100,100,50,100,25,100,10,100, and 5,100, to illustrate the performance of different algorithms in different-sized data. b=1,2,4,10, 20 for these five cases. A larger b indicates a more severely imbalanced dataset. Figure 2 shows the training datasets.Figure 2
Simulated Gaussian data for numeric studies.


## 5.1.2. Parameters Setup

The statistical analysis shows that the cost parametersc1 and c2 have to satisfy(48)c1≤pej1/2−pej≤pejϵjc−pej,(49)c2≤12ln2nc1−1≤12ln2∑ic11−yi/2−1≤12ln∑ic11−yi/2ϵjc−1.We set the upper boundaries as the chosen values ofc1 and c2 in (48) and (49). The restrictions of ϵjc<1/2 and c1≤1 indicate pej≤1/2ϵjc≤1/4. Equation (49) requires that 2nc1−1>0, which indicates c1>1/2n.We choose the values ofpej as 1/4,1/8,1/16,1/32,1/64, and 1/128 to ensure that both cost parameters have the values. Table 4 lists the cost parameters in the present five simulated datasets.Table 4
The cost parameter determinations.
pej1/41/81/161/321/641/128c111/31/71/151/311/63c2b=12.99452.44272.01401.62261.23840.8385b=22.85022.29761.86711.47221.08040.6625b=42.75872.20541.77361.37580.97750.5440b=102.69452.14061.70771.30750.90390.4566b=202.67122.11711.68361.28250.87670.4236


## 5.1.3. Evaluation Criteria

Twelve criteria listed in Table5 are used to evaluate the performance of algorithms in this paper, including the error rate criteria re and the misclassified cost criteria ce.Table 5
The criteria for classification evaluation.
Training Data Global Error Rateregtrain=yitrain≠yi∗train/nTraining data positive class error ratereptrain=yitrain≠yi∗train&yitrain=1/npTraining data global error costcegtrain=ci⋅yitrain≠yi∗train/∑iciSame size test data global error rateregtest=yitest≠yi∗test/nSame size test data positive class error ratereptest=yitest≠yi∗test&yitest=1/npSame size test data global error costcegtest=ci⋅yitest≠yi∗test/∑iciSmaller size test data global error rateregstest=yistest≠yi∗stest/nsSmaller size test data positive class error raterepstest=yistest≠yi∗stest&yistest=1/npsSmaller size test data global error costcegstest=ci⋅yistest≠yi∗stest/∑iciLarger size test data global error rateregltest=yiltest≠yi∗ltest/nlLarger size test data positive class error raterepltest=yiltest≠yi∗ltest&yiltest=1/nplLarger size test data global error costcegltest=ci⋅yiltest≠yi∗ltest/∑iciThe calculations ofregtest, reptest, and cegtest use the same-sized test data as the training data, ntrain=ntest=n,nptrain=nptest=np. Small-sized and large-sized test data are used for the calculations of regstest,repstest, and cegstest and regltest,repltest, and cegltest, respectively. It means ns<n<nl,nps<np<npl. Note that the positive class error cost is equal to the error rate since the positive cost parameter is set as 1. All the test datasets in this paper keep the same imbalance index b as the training data, namely,(50)S−testS+test=S−trainS+train=b.


## 5.1.4. Results of Simulation Studies

We run every simulated dataset 100 times and calculate the mean and standard deviation values as the results. Tables6–10 list the experiments. Obviously, with the imbalanced index b growing larger, the smaller cost parameters could better classify performances. We conjecture that the classification performance is best when the corresponding c1 is closest to 1/b.Table 6
The experiment results ofb=1.
c1=1c2=2.9945c1=1/3c2=2.4427c1=1/7c2=2.0140c1=1/15c2=1.6226c1=1/31c2=1.2384c1=1/63c2=0.8385regtrain0.0030(0.0438)0.0036 (0.0393)0.0035 (0.0310)0.0039 (0.0334)0.0040 (0.0554)0.0042 (0.0612)reptrain0.0031(0.0448)0.0034 (0.0329)0.0036 (0.0426)0.0039 (0.0381)0.0040 (0.0796)0.0041 (0.0540)cegtrain0.0030(0.0318)0.0034 (0.0538)0.0032 (0.0685)0.0035 (0.0317)0.0036 (0.0696)0.0038 (0.0745)regtest0.0031(0.0339)0.0031(0.0376)0.0032 (0.0657)0.0037 (0.0331)0.0039 (0.0300)0.0040 (0.0496)reptest0.0031 (0.0323)0.0036 (0.0537)0.0030(0.0849)0.0038 (0.0415)0.0039 (0.0338)0.0041 (0.0900)cegtest0.0031 (0.0408)0.0030(0.0644)0.0031 (0.0593)0.0035 (0.0595)0.0038 (0.0384)0.0039 (0.0563)regstest0.0031(0.0477)0.0033 (0.0333)0.0032 (0.0640)0.0037 (0.0446)0.0039 (0.0755)0.0042 (0.0620)repstest0.0031 (0.0332)0.0033 (0.0346)0.0030(0.0748)0.0039 (0.0536)0.0041 (0.0709)0.0042 (0.0657)cegstest0.0030(0.0439)0.0032 (0.0572)0.0031 (0.0554)0.0035 (0.0390)0.0036 (0.0420)0.0040 (0.0565)regltest0.0029(0.0323)0.0035 (0.0780)0.0031 (0.0437)0.0039 (0.0667)0.0040 (0.0434)0.00041 (0.0887)repltest0.0030(0.0478)0.0031 (0.0365)0.0032 (0.0776)0.0038 (0.0662)0.0040 (0.0835)0.0043 (0.0499)cegltest0.0029(0.0351)0.0034 (0.0522)0.0030 (0.0676)0.0037 (0.0334)0.0038 (0.0341)0.0040 (0.0344)Table 7
The experiment results ofb=2.
c1=1c2=2.9945c1=1/3c2=2.4427c1=1/7c2=2.0140c1=1/15c2=1.6226c1=1/31c2=1.2384c1=1/63c2=0.8385regtrain0.0043 (0.4673)0.0020(0.4204)0.0027 (0.4370)0.0026 (0.4246)0.0029 (0.3300)0.0033 (0.3555)reptrain0.0053 (0.3950)0.0026 (0.4368)0.0025(0.4637)0.0026 (0.4645)0.0026 (0.3891)0.0029 (0.4967)cegtrain0.0043 (0.4082)0.0022(0.3104)0.0024 (0.4686)0.0024 (0.4837)0.0027 (0.4853)0.0030 (0.3063)regtest0.0038 (0.3738)0.0021(0.4547)0.0035 (0.4430)0.0027 (0.3696)0.0029 (0.3119)0.0029 (0.4192)reptest0.0045 (0.3145)0.0028 (0.4728)0.0025(0.4966)0.0028 (0.3499)0.0031 (0.4755)0.0033 (0.4287)cegtest0.0038 (0.4071)0.0020(0.4748)0.0029 (0.4787)0.0027 (0.3140)0.0029 (0.4898)0.0029 (0.4157)regstest0.0042 (0.3660)0.0016(0.3660)0.0024 (0.3477)0.0029 (0.4636)0.0029 (0.4353)0.0031 (0.3849)repstest0.0055 (0.4836)0.0018(0.3095)0.0032 (0.3323)0.0032 (0.3091)0.0026 (0.4430)0.0031 (0.4331)cegstest0.0042 (0.3592)0.0025 (0.4214)0.0022(0.4206)0.0026 (0.4411)0.0028 (0.3444)0.0028 (0.3741)regltest0.0042 (0.3248)0.0018(0.3845)0.0026 (0.4646)0.0024 (0.4188)0.0028 (0.4434)0.0033 (0.3432)repltest0.0054 (0.4604)0.0020(0.3554)0.0026 (0.3905)0.0025 (0.3730)0.0031 (0.3185)0.0030 (0.3504)cegltest0.0042 (0.4394)0.0018(0.3538)0.0025 (0.3312)0.0023 (0.3973)0.0027 (0.3230)0.0031 (0.4799)Table 8
The experiment results ofb=5.
c1=1c2=2.9945c1=1/3c2=2.4427c1=1/7c2=2.0140c1=1/15c2=1.6226c1=1/31c2=1.2384c1=1/63c2=0.8385regtrain0.0062 (0.4142)0.0032 (0.3957)0.0025 (0.3406)0.0031 (0.2384)0.0032 (0.2024)0.0032 (0.2019)reptrain0.0089 (0.3637)0.0050 (0.4967)0.0030 (0.4904)0.0031 (0.3888)0.0031 (0.4530)0.0033 (0.2667)cegtrain0.0062 (0.2088)0.0030 (0.2824)0.0024 (0.4008)0.0030 (0.3674)0.0031 (0.2040)0.0031 (0.3920)regtest0.0061 (0.2904)0.0028 (0.2729)0.0026 (0.4036)0.0030 (0.4861)0.0030 (0.4195)0.0033 (0.4460)reptest0.0090 (0.4453)0.0058 (0.3510)0.0027 (0.3723)0.0033 (0.3545)0.0032 (0.2935)0.0033 (0.2351)cegtest0.0061 (0.3162)0.0027 (0.2519)0.0024 (0.4383)0.0029 (0.3922)0.0028 (0.3349)0.0031 (0.2821)regstest0.0063 (0.4622)0.0028 (0.4731)0.0029 (0.4915)0.0033 (0.4227)0.0031 (0.3523)0.0031 (0.3897)repstest0.0093 (0.4590)0.0051 (0.4006)0.0025 (0.3267)0.0033 (0.3091)0.0032 (0.3934)0.0033 (0.2366)cegstest0.0063 (0.4888)0.0027 (0.3809)0.0027 (0.3463)0.0030 (0.2130)0.0030 (0.4935)0.0028 (0.4646)regltest0.0068 (0.3685)0.0032 (0.2169)0.0026 (0.2913)0.0030 (0.4536)0.0032 (0.4427)0.0031 (0.3334)repltest0.0088 (0.4268)0.0050 (0.2679)0.0030 (0.3359)0.0033 (0.2583)0.0031 (0.3136)0.0031 (0.2392)cegltest0.0068 (0.2535)0.0029 (0.3750)0.0024 (0.2735)0.0028 (0.3600)0.0030 (0.3842)0.0030 (0.4166)Table 9
The experiment results ofb=10.
c1=1c2=2.9945c1=1/3c2=2.4427c1=1/7c2=2.0140c1=1/15c2=1.6226c1=1/31c2=1.2384c1=1/63c2=0.8385regtrain0.2131 (0.3640)0.0629 (0.3041)0.0029 (0.5293)0.0028(0.3091)0.0029 (0.3091)0.0029 (0.3396)reptrain0.4229 (0.5814)0.2629 (0.4365)0.0029 (0.4660)0.0028(0.5930)0.0028(0.5930)0.0031 (0.5406)cegtrain0.2030 (0.4871)0.0630 (0.5596)0.0028(0.5697)0.0030 (0.3245)0.0030 (0.3245)0.0030 (0.5110)regtest0.2533 (0.2023)0.0630 (0.4073)0.0030 (0.3995)0.0028 (0.2032)0.0028 (0.2032)0.0027(0.4745)reptest0.4531 (0.5172)0.2630 (0.3888)0.0028(0.3106)0.0030 (0.4138)0.0030 (0.4138)0.0028(0.4054)cegtest0.2629 (0.5811)0.0631 (0.4503)0.0029 (0.5217)0.0028(0.4158)0.0028(0.4158)0.0033 (0.2670)regstest0.2530 (0.4356)0.0629 (0.4585)0.0030(0.3506)0.0030(0.2349)0.0030(0.2349)0.0033 (0.4934)repstest0.4230 (0.3356)0.2629 (0.2448)0.0029(0.5382)0.0029(0.3647)0.0030 (0.3647)0.0033 (0.5974)cegstest0.2231 (0.3216)0.0630 (0.3403)0.0030 (0.5239)0.0028(0.3297)0.0028(0.3297)0.0033 (0.4633)regltest0.3029 (0.3406)0.0629 (0.4907)0.0028(0.3624)0.0029 (0.5111)0.0029 (0.5111)0.0030 (0.3396)repltest0.4031 (0.2818)0.2629 (0.2790)0.0029 (0.2387)0.0028(0.5565)0.0029 (0.5565)0.0030 (0.3725)cegltest0.3030 (0.5284)0.0629 (0.5398)0.0029 (0.3888)0.0027(0.4037)0.0027(0.4037)0.0028 (0.5488)Table 10
The experiment results ofb=20.
c1=1c2=2.9945c1=1/3c2=2.4427c1=1/7c2=2.0140c1=1/15c2=1.6226c1=1/31c2=1.2384c1=1/63c2=0.8385regtrain0.2032 (0.2626)0.1233 (0.3151)0.0829 (0.4914)0.0030 (0.5119)0.0029 (0.4911)0.0033 (0.5357)reptrain0.5730 (0.2680)0.3132 (0.4644)0.1330 (0.3304)0.0026 (0.5655)0.0026 (0.2402)0.0029 (0.3079)cegtrain0.2033 (0.2633)0.1232 (0.4608)0.0833 (0.4054)0.0030 (0.2351)0.0027 (0.2147)0.0030 (0.5034)regtest0.2236 (0.5247)0.1729 (0.3532)0.0817 (0.3587)0.0027 (0.5364)0.0029 (0.5309)0.0029 (0.4907)reptest0.5432 (0.3757)0.3730 (0.5249)0.1360 (0.2709)0.0028 (0.4515)0.0031 (0.2166)0.0033 (0.4749)cegtest0.2733 (0.2104)0.1731 (0.3326)0.0829 (0.3310)0.0027 (0.4997)0.0029 (0.3874)0.0029 (0.4969)regstest0.2830 (0.3424)0.1632 (0.4303)0.0932 (0.5882)0.0029 (0.4927)0.0029 (0.4914)0.0031 (0.5735)repstest0.5729 (0.5996)0.3829 (0.5110)0.2532 (0.5348)0.0032 (0.4090)0.0026 (0.4582)0.0031 (0.3844)cegstest0.2931 (0.2220)0.1531 (0.2105)0.0866 (0.3779)0.0026 (0.2612)0.0030 (0.4546)0.0028 (0.5995)regltest0.2429 (0.2612)0.1733 (0.2488)0.0832 (0.5526)0.0024 (0.2521)0.0028 (0.4023)0.0033 (0.2909)repltest0.6031 (0.2359)0.4029 (0.3856)0.2772 (0.3256)0.0025 (0.5423)0.0031 (0.5677)0.0030 (0.2734)cegltest0.2329 (0.2745)0.1631 (0.4322)0.0830 (0.3358)0.0031 (0.4016)0.0027 (0.2118)0.0031 (0.4080)


## 5.2. Real Data Studies

Four real binary datasets are studied, and the values of their imbalanced indexb are 15.13, 11.59, 5.14, and 3.36. Since all the datasets are imbalanced, each dataset is randomly split in half as the training part and half as the test part to avoid the situation that there are no or too few positive samples in the training or test data. We choose thirteen popular classification algorithms in Table 11 for comparisons. The first five are all the traditional ones and the sixth, PRVM [48], is designed for the imbalanced data problem, which has achieved convincing results. The last seven ones are the traditional cost-sensitive classifiers, which we have discussed in the discussion part. Note that the AdaImC algorithm in this part is tested based on the different determinations of the cost factors to choose the optimal parameter setup before joining the comparison. We will not show this process again in the real data studies.Table 11
Compared machine learning algorithms in real data studies.
NameDescriptionParametersSVMsStandard single support vector machines with RBF kernelγbest is obtained by grid searching in 2−20:20Ada-DTAdaBoost using decision tree as weak classifiersAda-LSVMsAdaBoost using linear kernel SVMs as weak classifiersAda-RSVMsAdaBoost using RBF kernel SVMs as weak classifiersγbest is obtained by grid searching in 2−20:20RVMStandard single relevance vector machines with RBF kernelγbest is obtained by grid searching in 2−20:20PRVMRelevance vector machines under the probit link function with RBF kernelγbest is obtained by grid searching in 2−20:20ICETIncorporates misclassification costs in the fitness function of genetic algorithmsCost-sensitive decision treesUse the misclassification costs in tree building processMetaCostConvert cost-insensitive classifiers into cost-sensitive ones without modifyingCost-sensitive naive BayesUse cost factor information as the priorEmpirical thresholdingUse bagging on decision trees to obtain reliable probability estimationsCostingUse rejection sampling to resample the dataWeightingAssign a normalized weight to each instance according to the misclassification costs


### 5.2.1. Seismic-Bumps Data Study

The seismic-bumps data withb=15.13 are obtained from the UCI Machine Learning Repository [49]. This dataset is about the seismic hazard in mining activity. Mining activity is always connected with the occurrence of dangers, which are called mining hazards. Seismic hazard is a special and dangerous case in mining hazards. The seismic-bumps dataset is intensely imbalanced with only 170 samples of the positive class among a total of 2584 samples, and it contains 18 explanatory variables, including seismic hazard assessment obtained by different methods, seismic energy recorded by different types of equipment, and the number of seismic bumps in different energy range. The hazardous state is the negative class defined as a high energy seismic bump occurring in the next shift, which is indicated by y=−1. The nonhazardous state is the positive class defined as no high energy seismic bump occurring in the next shift, which is indicated by y=1.Table12 displays the seismic-bumps data experiment. It shows the great performance of our proposed algorithms for the classification of this dataset. Except for the specificity, all evaluation criteria of the PRVM and the AdaImC perform better than other algorithms. The AdaImC obviously outperforms the PRVM except for the F−1Measure.Table 12
The results of seismic-bumps datab=15.13.
AlgorithmsEvaluation measuresSensitivitySpecificityF-1 MeasureAUCGlobal AccuracySVMs0.00001.00000.00000.50.9320Ada-DT0.10470.98730.16360.54600.9273Ada-LSVMs0.00000.99920.00000.49960.9312Ada-RSVMs0.62790.66750.20300.64770.6648RVM0.53180.97550.86010.21010.6533PRMV0.90110.93640.96010.91010.9473ICET0.91130.95780.92210.91220.9361Cost-sensitive decision trees0.88040.93820.91300.91430.9473MetaCost0.90430.94610.93160.90060.9422Cost-sensitive naive Bayes0.90180.94180.90880.92000.9461Empirical thresholding0.89330.96550.92870.91550.9327Costing0.91610.92850.91290.92670.9378Weighting0.90650.94990.90160.93310.9502AdaImC0.92770.98530.91120.93760.9588


### 5.2.2. Glass-2 Data Study

The Glass-2 dataset is obtained from the Knowledge Extraction based on Evolutionary Learning (KEEL) Dataset Repository [50]. This dataset is originally from the US Forensic Science Service and intensely imbalanced with only 17 samples of the positive class among a total of 214 samples, which means b=11.59. The Glass-2 dataset has eight explanatory variables, including the refractive index and the different chemical elements content. The positive class is labeled as the glass made by the nonfloat process and indicated by y=1. The negative class is defined as the glass made by other processed methods and indicated by y=−1.Table13 displays the Glass-2 data experiment. It shows similar results as the seismic-bumps data. Except for the Specificity, all evaluation measures of PRVM and AdaImC are better than other algorithms. Compared to the PRVM, our proposed AdaImC has a slight improvement.Table 13
The results of Glass-2 datab=11.59.
AlgorithmsEvaluation measuresSensitivitySpecificityF-1 MeasureAUCGlobal AccuracySVMs0.00001.00000.00000.50.8972Ada-DT0.09090.98960.15380.54020.8972Ada-LSVMs0.00001.00000.00000.50000.8970Ada-RSVMs0.18180.90630.18180.54400.8318RVM0.14330.95880.17100.60310.8313PRMV0.92670.90880.94330.92760.9006ICET0.91330.95260.93760.92730.9361Cost-sensitive decision trees0.90630.97270.92880.91880.9422MetaCost0.91220.95280.94760.93810.9418Cost-sensitive naive Bayes0.90050.93170.97020.94270.9355Empirical thresholding0.89730.94660.94660.93290.9341Costing0.91210.97520.93100.94040.9362Weighting0.90330.94990.93410.93870.9132AdaImC0.92930.90190.95180.97860.9470


### 5.2.3. New-Thyroid-1 Data Study

The new-thyroid-1 dataset is also obtained from the KEEL Dataset Repository. This dataset is originally from the Garvan Institute in Sydney, Australia. It includes the information of thyroid patients and is rearranged by KEEL to be an imbalanced binary dataset. The new-thyroid-1 dataset is imbalanced with 35 positive samples among a total of 215 samples, which indicates the imbalanced index isb=5.14. This dataset contains 5 explanatory variables, including the levels of different hormones such as thyroxin and triiodothyronine. The positive class is labeled as hyperthyroidism, which is indicated by y=1. The negative class is defined as nonhyperthyroidism, which is indicated by y=−1.Table14 displays the results of the new-thyroid-1 data experiment. It shows the satisfactory performance of our proposed algorithm compared to others for this data classification problem. Almost all the evaluation criteria of the PRVM and the AdaImC perform better than other algorithms. Because b=5.14 is a relatively small number in this dataset, the differences between PRVM and AdaImC are not significant.Table 14
The results of new-thyroid-1 datab=5.14.
AlgorithmsEvaluation measuresSensitivitySpecificityF-1 MeasureAUCGlobal AccuracySVMs0.26321.00000.41670.63160.8704Ada-DT0.78950.98880.85710.88910.9537Ada-LSVMs0.84210.98880.88890.91540.9630Ada-RSVMs0.84210.96630.84210.90420.9444RVM0.83110.94720.87320.90310.9135PRMV0.91330.98460.95720.97310.9655ICET0.91720.96790.92210.91220.9361Cost-sensitive decision trees0.89540.97550.91300.91430.9473MetaCost0.90070.94210.93160.90060.9422Cost-sensitive naive Bayes0.91100.90490.90880.92000.9461Empirical thresholding0.88940.94820.92870.91550.9327Costing0.91610.95620.93310.92670.9378Weighting0.90480.95730.90160.93310.9502AdaImC0.91480.98330.96200.97440.9670


### 5.2.4. Ecoli-1 Data Study

The Ecoli-1 dataset is also obtained from the KEEL Dataset Repository. This dataset initially comes from the Institute of Molecular and Cellular Biology at Osaka University. It includes the information of the cellular localization sites of proteins and is rearranged by KEEL as an imbalanced binary dataset. The Ecoli-1 dataset is imbalanced with 77 positive samples among a total of 336 samples, which indicates the imbalanced index isb=3.36. This dataset contains seven explanatory variables, including the signal sequence recognition scores by different methods and the presence of charge on N-terminus. The positive class is labeled as the inner membrane without a signal sequence, which is indicated by y=1. The negative class is defined as other situations, which is indicated by y=−1.Table15 displays the results of the Ecoli-1 data experiment. It shows all the algorithms perform similarly since the imbalanced situation is not obvious. Our proposed AdaImC still has a convincing performance for this balanced dataset.Table 15
The results of Ecoli-1 datab=3.36.
AlgorithmsEvaluation measuresSensitivitySpecificityF-1 MeasureAUCGlobal AccuracySVMs0.88890.92420.82050.90660.9167Ada-DT0.86110.90910.78480.88510.8988Ada-LSVMs0.00001.00000.00000.50000.7857Ada-RSVMs0.94440.86360.77270.90400.8810RVM0.95310.91430.83310.92120.9145PRMV0.97790.94310.98220.92210.9055ICET0.94660.95880.96420.90670.8730Cost-sensitive decision trees0.95120.98310.95800.91220.8933MetaCost0.94880.97930.97310.90520.8517Cost-sensitive naive Bayes0.95710.96830.97380.91800.7841Empirical thresholding0.93700.96810.96390.90330.8905Costing0.95190.94880.94880.90730.9022Weighting0.96110.97690.95810.91800.8467AdaImC0.96440.93550.98830.92750.9108Our proposed AdaImC algorithm outperforms the traditional ones in imbalanced data classification based on real data studies. With a large imbalanced indexb, our proposed algorithm shows excellent performance.


## 5.2.1. Seismic-Bumps Data Study

The seismic-bumps data withb=15.13 are obtained from the UCI Machine Learning Repository [49]. This dataset is about the seismic hazard in mining activity. Mining activity is always connected with the occurrence of dangers, which are called mining hazards. Seismic hazard is a special and dangerous case in mining hazards. The seismic-bumps dataset is intensely imbalanced with only 170 samples of the positive class among a total of 2584 samples, and it contains 18 explanatory variables, including seismic hazard assessment obtained by different methods, seismic energy recorded by different types of equipment, and the number of seismic bumps in different energy range. The hazardous state is the negative class defined as a high energy seismic bump occurring in the next shift, which is indicated by y=−1. The nonhazardous state is the positive class defined as no high energy seismic bump occurring in the next shift, which is indicated by y=1.Table12 displays the seismic-bumps data experiment. It shows the great performance of our proposed algorithms for the classification of this dataset. Except for the specificity, all evaluation criteria of the PRVM and the AdaImC perform better than other algorithms. The AdaImC obviously outperforms the PRVM except for the F−1Measure.Table 12
The results of seismic-bumps datab=15.13.
AlgorithmsEvaluation measuresSensitivitySpecificityF-1 MeasureAUCGlobal AccuracySVMs0.00001.00000.00000.50.9320Ada-DT0.10470.98730.16360.54600.9273Ada-LSVMs0.00000.99920.00000.49960.9312Ada-RSVMs0.62790.66750.20300.64770.6648RVM0.53180.97550.86010.21010.6533PRMV0.90110.93640.96010.91010.9473ICET0.91130.95780.92210.91220.9361Cost-sensitive decision trees0.88040.93820.91300.91430.9473MetaCost0.90430.94610.93160.90060.9422Cost-sensitive naive Bayes0.90180.94180.90880.92000.9461Empirical thresholding0.89330.96550.92870.91550.9327Costing0.91610.92850.91290.92670.9378Weighting0.90650.94990.90160.93310.9502AdaImC0.92770.98530.91120.93760.9588


## 5.2.2. Glass-2 Data Study

The Glass-2 dataset is obtained from the Knowledge Extraction based on Evolutionary Learning (KEEL) Dataset Repository [50]. This dataset is originally from the US Forensic Science Service and intensely imbalanced with only 17 samples of the positive class among a total of 214 samples, which means b=11.59. The Glass-2 dataset has eight explanatory variables, including the refractive index and the different chemical elements content. The positive class is labeled as the glass made by the nonfloat process and indicated by y=1. The negative class is defined as the glass made by other processed methods and indicated by y=−1.Table13 displays the Glass-2 data experiment. It shows similar results as the seismic-bumps data. Except for the Specificity, all evaluation measures of PRVM and AdaImC are better than other algorithms. Compared to the PRVM, our proposed AdaImC has a slight improvement.Table 13
The results of Glass-2 datab=11.59.
AlgorithmsEvaluation measuresSensitivitySpecificityF-1 MeasureAUCGlobal AccuracySVMs0.00001.00000.00000.50.8972Ada-DT0.09090.98960.15380.54020.8972Ada-LSVMs0.00001.00000.00000.50000.8970Ada-RSVMs0.18180.90630.18180.54400.8318RVM0.14330.95880.17100.60310.8313PRMV0.92670.90880.94330.92760.9006ICET0.91330.95260.93760.92730.9361Cost-sensitive decision trees0.90630.97270.92880.91880.9422MetaCost0.91220.95280.94760.93810.9418Cost-sensitive naive Bayes0.90050.93170.97020.94270.9355Empirical thresholding0.89730.94660.94660.93290.9341Costing0.91210.97520.93100.94040.9362Weighting0.90330.94990.93410.93870.9132AdaImC0.92930.90190.95180.97860.9470


## 5.2.3. New-Thyroid-1 Data Study

The new-thyroid-1 dataset is also obtained from the KEEL Dataset Repository. This dataset is originally from the Garvan Institute in Sydney, Australia. It includes the information of thyroid patients and is rearranged by KEEL to be an imbalanced binary dataset. The new-thyroid-1 dataset is imbalanced with 35 positive samples among a total of 215 samples, which indicates the imbalanced index isb=5.14. This dataset contains 5 explanatory variables, including the levels of different hormones such as thyroxin and triiodothyronine. The positive class is labeled as hyperthyroidism, which is indicated by y=1. The negative class is defined as nonhyperthyroidism, which is indicated by y=−1.Table14 displays the results of the new-thyroid-1 data experiment. It shows the satisfactory performance of our proposed algorithm compared to others for this data classification problem. Almost all the evaluation criteria of the PRVM and the AdaImC perform better than other algorithms. Because b=5.14 is a relatively small number in this dataset, the differences between PRVM and AdaImC are not significant.Table 14
The results of new-thyroid-1 datab=5.14.
AlgorithmsEvaluation measuresSensitivitySpecificityF-1 MeasureAUCGlobal AccuracySVMs0.26321.00000.41670.63160.8704Ada-DT0.78950.98880.85710.88910.9537Ada-LSVMs0.84210.98880.88890.91540.9630Ada-RSVMs0.84210.96630.84210.90420.9444RVM0.83110.94720.87320.90310.9135PRMV0.91330.98460.95720.97310.9655ICET0.91720.96790.92210.91220.9361Cost-sensitive decision trees0.89540.97550.91300.91430.9473MetaCost0.90070.94210.93160.90060.9422Cost-sensitive naive Bayes0.91100.90490.90880.92000.9461Empirical thresholding0.88940.94820.92870.91550.9327Costing0.91610.95620.93310.92670.9378Weighting0.90480.95730.90160.93310.9502AdaImC0.91480.98330.96200.97440.9670


## 5.2.4. Ecoli-1 Data Study

The Ecoli-1 dataset is also obtained from the KEEL Dataset Repository. This dataset initially comes from the Institute of Molecular and Cellular Biology at Osaka University. It includes the information of the cellular localization sites of proteins and is rearranged by KEEL as an imbalanced binary dataset. The Ecoli-1 dataset is imbalanced with 77 positive samples among a total of 336 samples, which indicates the imbalanced index isb=3.36. This dataset contains seven explanatory variables, including the signal sequence recognition scores by different methods and the presence of charge on N-terminus. The positive class is labeled as the inner membrane without a signal sequence, which is indicated by y=1. The negative class is defined as other situations, which is indicated by y=−1.Table15 displays the results of the Ecoli-1 data experiment. It shows all the algorithms perform similarly since the imbalanced situation is not obvious. Our proposed AdaImC still has a convincing performance for this balanced dataset.Table 15
The results of Ecoli-1 datab=3.36.
AlgorithmsEvaluation measuresSensitivitySpecificityF-1 MeasureAUCGlobal AccuracySVMs0.88890.92420.82050.90660.9167Ada-DT0.86110.90910.78480.88510.8988Ada-LSVMs0.00001.00000.00000.50000.7857Ada-RSVMs0.94440.86360.77270.90400.8810RVM0.95310.91430.83310.92120.9145PRMV0.97790.94310.98220.92210.9055ICET0.94660.95880.96420.90670.8730Cost-sensitive decision trees0.95120.98310.95800.91220.8933MetaCost0.94880.97930.97310.90520.8517Cost-sensitive naive Bayes0.95710.96830.97380.91800.7841Empirical thresholding0.93700.96810.96390.90330.8905Costing0.95190.94880.94880.90730.9022Weighting0.96110.97690.95810.91800.8467AdaImC0.96440.93550.98830.92750.9108Our proposed AdaImC algorithm outperforms the traditional ones in imbalanced data classification based on real data studies. With a large imbalanced indexb, our proposed algorithm shows excellent performance.


## 5.3. Algorithm Complexity Analysis

Our proposed AdaImC algorithm outperforms the classical AdaBoost, especially in the imbalanced data problem. However, it has a more complex structure, which means the model complexity is larger than the classic one. In practice, especially for big data, our proposed one may face an inefficiency issue.Assume thatc1 has been determined. Compared with the classic AdaBoost (Algorithm 1), the AdaImC (Algorithm 4) has the following additional computation for the training dataset S:Step 0: “n+2n,” where“n” is the calculated amount for Dti=c11−yi/2 and “2n” is the calculated amount for the(51)0<c2<12ln∑i:htxi=yiDitc11−yi/2∑i:htxi≠yiDitc11−yi/2.Step 2: “2n” is the calculated amount for(52)αt=12ln∑i:htxi=yiDitc11−yi/2∑i:htxi≠yiDitc11−yi/2−c2.Step 3: “3n” is the calculated amount for(53)Dt+1i=c11−yi/2Dtiexp−yihtxiαt+c2n∏t=1TZt.Totally, AdaImC has the additional8n computations than the classic AdaBoost, which seems acceptable for the practice. We compare the code running time for AdaImC and the classic AdaBoost based on the above four real datasets in Table 16. In relatively big datasets, the AdaImC needs significantly more time in practice. The additional time for code running is acceptable, and the complexity does not lower the usability of our proposed algorithm based on this limited study.Table 16
The results of algorithms’ runningTimea.
AlgorithmsCode running time (seconds)Seismic-bumpsGlass-2New-thyroid-1Ecoli-1AdaBoost54.3336.1229.8027.00AdaImC61.7340.5131.2232.91Increase percentage13.62%12.15%4.77%21.89%aPerformed in Python 3.6 on macOS 10.12.6.


## 6. Conclusion and Discussion

The cost-sensitive learning model and the AdaBoost algorithm are two main approaches in imbalanced data classification. The combination of them, namely, cost-sensitive AdaBoost, is receiving more attention. Nevertheless, the challenging problem is that almost all cost-sensitive boosting algorithms have no theoretic proof, although they perform nicely in practice. The present article tries to solve this problem by studying the cost-sensitive AdaBoost algorithms for the imbalanced data classification with two contributions. First, we summarize most cost-sensitive AdaBoost algorithms in the literature, which are all heuristic models without theoretic proof. A comprehensively general form is proposed, and we set a particular one, namely, the AdaImC algorithm. Our proposed model considers adjusting the weight parameter of weak classifiers and the weight parameter of the data samples in the AdaBoost algorithm with the cost factors. A theoretic proof of boosting property is given to support our algorithm. Second, we study the PoE (Product of Experts) model in statistics. We conclude that our proposed AdaImC algorithm is reasonable since a particular form of the PoE model is identical to the AdaImC algorithm. Besides, the PoE model provides a deeper insight into the relationship between the cost parameters, which helps us determine the values of cost factors more logically.More studies for cost-sensitive AdaBoost are ponderable in the future. The spontaneous idea extends to the multiple classes’ problem, which has a more significant practical effect. The present article only considers a particular form for the cost-sensitive AdaBoost, but we can focus on the general form and provide the theoretical proofs from machine learning and statistics perspectives. Also, the choices of the values of the cost parameters are limited in this paper. A broader range of the values of parameters would provide more convincing results. Another point worth mentioning is that the proposed model is not robust under seriously imbalanced datasets with imbalanced index larger than 20, where the outlier detection algorithms are more suitable.



---

*Source: 3165589-2021-10-23.xml*