Imbalanced Data in Classification: General Solution & Case Study | by Hadeer Hammad | Towards Data Science

Authors: Hadeer Hammad & Min Zhou, PhD
When dealing with a classification problem, how does the one approach a dataset that has significantly more data points in one of its binary classes compared to the other class? Such datasets are referred as imbalanced. Would it be wise to simply implement a classification model on the imbalanced data and wish for the best? There are many resampling techniques out there that “balance” imbalanced datasets and plenty of classification models. How do we select the optimal one for our problem?
The intention of this article is to summarize the following academic research “Imbalanced Classification: an objective-oriented review” for non-technical practitioners in the data science field [https://arxiv.org/abs/2002.04592]. This article is accessible to readers with at minimum basic prior knowledge in Machine Learning.
Imbalanced datasets are very common across all industries. Let’s assume you work for a bank and you have a credit-card transaction data with fraud labels. You discover that the fraud class is highly imbalanced (0.17% of data is Fraud=0 & 99.83% of data is Non_Fraud=1). Naively implementing a classification model on such imbalanced data will likely result in very low predictive accuracy. In classification problems, it is almost impossible to do a perfect job in simultaneously predicting the two classes (Fraud and Non-Fraud). This is because classification models carry two types of errors, type I and type II, and inherently there is a trade-off between the two. Type I error indicates the probability of misclassifying a fraud transaction as non-fraud. Type II error means the opposite: the probability of misclassifying a non-fraud transaction as fraud. It is often the case that a Type I error is more costly than a Type II error. Our case here is an example of that, as you could anger your clients by wrongly flagging their transactions to be fraud and thereby risk losing them. However, generally speaking, depending on the nature of the problem, you may choose to treat the weights of type I and type II errors differently, which accordingly determines the classification paradigm to operate under.
We will provide a brief overview of classification paradigms and resampling tecnhiques. Then, we will use the above example of credit-card fraud data, which is a real case, to pair up each resampling technique with classification models under different classification paradigms. The objective is to find the optimal combination of resampling technique with classification model for each paradigm. At the end of this article, we provide samples of writing R codes on applying classification paradigms and resampling techniques.
Each classification paradigm treats the weights of type I and type II errors differently:
1. Classical Classification (CC) paradigm minimizes the overall classification error, which implies equal weights on type I and type II errors. It does not serve well in cases where type I error could be more serious than type II error or vice-versa. Thus, we will next introduce two other paradigms that address this issue of asymmetric error importance.
2. Cost-Sensitive (CS) learning paradigm assigns different costs to type I and type II errors and then minimizes their totality. It has multiple approaches, but we will apply the postprocessing method onto our case study. The downside of CS learning is assigning cost values to type I and type II errors, as they are usually unknown. Another disadvantage is that even if we tune the empirical type I error to be equal to a pre-specified level (alpha), the true population-level type I error still has some likelihood of exceeding alpha. Nevertheless, Neyman-Pearson deals with this issue.
3. Neyman-Pearson paradigm is an emerging statistical framework that controls asymmetric error through aiming to minimize type II error while controlling type I error under a desirable level. To learn more, please read the following paper: Neyman-Pearson classification algorithms and NP receiver operating characteristics
We leverage resampling techniques to alleviate the effect of the imbalanced class size on the predictive power of classification models. Resampling techniques create new training datasets through balancing the number of data points in the minority and majority classes. We will go over three different techniques: undersampling, oversampling, and hybrid.
1. Undersampling method eliminates a subset of data points in the majority class, using two general approaches. The first one is random-undersampling, which randomly discards data points from the majority class to balance its size with that of the minority class. While it is a simple and time-efficient approach, it could create bias in the data by throwing information that makes the data sample less representative of the true entire population. The second technique, Cluster-based undersampling, could weaken this bias issue. It applies a clustering algorithm on the majority class in order to obtain a few clusters which are balanced with the size of the minority class. This technique could be slow if the size of the majority class is large. A general drawback of undersampling is that it could cause critical information to be lost given that a large portion of the majority class’s data is being discarded.
2. Oversampling increases the number of data points in the minority class to balance it with the size of the majority class. Random oversampling achieves this through randomly duplicating data points in the minority class until they are equally proportionate to the size of the majority class. Synthetic Minority Over-sampling Technique (SMOTE) utilizes k nearest neighbors to generate new data points for the minority class that are equally proportionate to the size of the majority class. Oversampling method could lead to over-fitting and it usually requires longer training time relative to undersampling.
3. Hybrid is a mere combination of undersampling and oversampling techniques. It simultaneously applies oversampling to increase the number of data points in the minority class, while applying undersampling to lower the number of data points in the majority class.
Dataset source: Kaggle [https://www.kaggle.com/mlg-ulb/creditcardfraud]
Number of Predictors: 30
Number of records/transactions: 284,807
Response variable: “Class,” which takes value 1 in case of fraud and 0 otherwise. In our data processing, we have reversed the label in our study, so if fraud, class 0; otherwise 1.
Imbalance ratio: 0.17% (492 fraudulent transactions out of 284,807, 492 ÷ 284807)
Due to our limited computational power, we extract a subsample from the above large dataset. We specify the imbalance ratio (IR) as 100; that is, IR=100. We randomly select n0=300 data points for class 0 (fraud) and select n1=n0*IR=30,000 for class 1. This creates our training set. The testing set consists of the remaining m0=192, [492–300] for class 0 and m1=m0*IR=19,200 for class 1. This splitting mechanism guarantees that IR will be equal for the training and testing sets.
Below are the resampling techniques that we use
No resampling: using the training set without making any modifications to the original data (in the charts, we refer to it as “Original”)
Random undersampling (we call it “Under”)
Oversampling (“SMOTE”): we use R Package smotefamliy, v1.3.1, Siriseriwan 2019 with the default choice of nearest neighbors
Hybrid: combining random undersampling with SMOTE
The following are classification models that we apply, followed by R packages (note: we use the packages’ default parameters):
Logistic regression (glm function in base R)
Support vector machine (R Package e1071, v1.7.2, Meyer et al. 2019)
XGBoost (R Package xgboost, v0.90.0.2, Chen et al. 2019)
For classification paradigms, we specify the following parameters:
CS learning paradigm: we specify the cost C0=IR=100 and C1=1; threshold=C0/(C0+C1)=100/101
NP paradigm: we specify α = 0.05 and violation rate δ = 0.05
Denote |S| as the cardinality of a set S. Let O = {CC, CS, NP}, T = {Original, Under, SMOTE, Hybrid}, C = {LR, SVM, XGB}. Hence, it entails |O=3| × |T=4| × |C=3| = 36 elements as illustrated below in Figure 1 which we will examine later.
Before assessing our results, we need to first set the foundation by reviewing several evaluation metrics, which are used to assess the models’ performance. Please note that we label the minority class (Fraud) as 0 and the majority class (No Fraud) as 1. We call class 0 (negative class, Fraud) and class 1 (positive class, Fraud).
Classification models are typically summarized in a table called confusion matrix, such as the one below. It is called “confusion” because classification models are never perfect in identifying the positive from the negative class (that is, they confuse them). The right diagonal represents the labels that were wrongly predicted, while the left one represents the labels that were correctly predicted.
Confusion matrix comprises of the following:
True Negative (TN): the number of data points that were correctly predicted to be negative (fraud)
False Negative (FN): the number of data points that were incorrectly predicted to be negative
True Positive (TP): the number of data points that were correctly predicted to be positive (no fraud)
False Positive (FP): the number of data points that were incorrectly predicted to be positive
— Type I error: the proportion of misclassifying a fraud transaction as non-fraud:= FP/(TN+FP)= 22/(170+22)=0.1146
— Type II error: the proportion of misclassifying a non-fraud transaction as fraud: = FN/(FN+TP)=477(477+18723)=0.0248
Below we compute the proportion of each class to use that information when computing empirical risk and classification cost next:
— Overall Classification error (or “empirical risk”): the percentage of misclassifying data points in both classes. It is denoted as follows:=Prop0*Type I + Prop1*Type II =(FP+FN)/(TP+FP+TN+FN)=0.0257
— In the case of assigning costs to type I and type II errors (C0=100 and C1=1, respectively), the empirical classification cost would be computed as follows:= C0*Prop0*Type1+C1*TypeII=100*0.0099*0.1146+1*0.9901*0.0248= 0.1380
— F-score is a common metric when dealing with imbalanced classification problems. It is derived from measuring precision and recall; please see the following table 3 for their computations. F-score is set to 0 if the corresponding precision or recall is undefined or equal to 0.
o Precision: measures the number of times your model made correct predictions out of its predicted identifications (that is, correct and incorrect predictions) for each class.
o Recall: measures the number of times your model made correct predictions out of the true identifications for each class.
Using the above computations of precision and recall, we calculate the F-score for each class:
The Receiver Operating Characteristic (ROC) curve is a graphical illustration of the classification model’s performance in identifying the positive from the negative class as the discrimination threshold is varied (see below Figure 2). Additionally, it reflects the trade-off between type I and type II errors. As we successfully identify more true negative’s (Fraud) in our data, it comes at the expense of misidentifying non-fraud records as fraud (committing false positive). The area under the ROC curve (AUC) is a single-number summary that quantifies the model’s performance in discriminating the positive from the negative class. It typically falls between 0 and 1. The larger the value, the better an indication of the models’ performance. A value of 0.5 does not serve any better than making a random guess. As depicted below in figure 2, ROC-AUC is 0.9734907, which suggests that the model has done a nice job on identifying fraud transactions from non-fraud.
Precision-Recall (PR) curves and their AUCs (PR-AUC) are alternative metrics when dealing with imbalanced data. In Figure 3, PR-AUC is 0.857943 for Fraud class and 0.9995953 for non-fraud class.
In summary, for type I and type II errors, empirical risk, and empirical classification costs, smaller values indicate better classification models’ performance. However, for F-score’s, ROC-AUC, and PR-AUC, larger values indicate better performance.
To stabilize our results, the process is repeated 50 times and the average performance is reported (the mean of evaluation metrics) for all classifiers under the three classification paradigms. The results are summarized in Figures 4 to 7 and Tables 4 to 6.
Figure 4 (above) exhibits the results of pairing up each resampling technique with classification models under CC paradigm. We observe that the empirical risk of all classifiers without resampling is smaller (better) than that of all resampling techniques. Additionally, F-scores are also better (higher) for all classifiers without resampling techniques. Thus, we conclude that if our learning objective is to minimize risk (operating under CC paradigm), applying XGBoost classification model on the original data without performing resampling technique is the optimal choice (“XGB+Original”).
It is worthy to note that minimizing the overall classification error on imbalanced data could lead to large type I error, which is clearly shown above. Nevertheless, we can have much better control over type I error through leveraging resampling techniques.
For readers’ convenience, we summarize below the optimal combinations of resampling techniques with classification models under CC paradigm according to several evaluation metrics:
ROC-AUC and PR-AUCs in Figure 5 (below) measure the overall models’ performance without the need to specify the classification paradigm. Those metrics suggest that XGBoost works best across all resampling techniques. Nonetheless, resampling techniques cannot improve the performance of logistic regression according to ROC-AUC. However, they can benefit support vector machine significantly under ROC-AUC and PR-AUC (Non-Fraud class). Finally, “XGB+Hybrid” is the optimal combination by the standards of ROC-AUC and PR-AUC (Non-Fraud class), while “XGB+SMOTE” is the best one by the assessment of PR-AUC (Fraud class).
If our objective is to minimize the total misclassification cost, then we should operate under CS learning paradigm. From the above figure 6, we discover that all the models benefit significantly from resampling techniques by using type I error as the evaluation metric. The table below summarizes the optimal combinations of resampling techniques with classification models under CS learning paradigm by the assessment of different metrics.
If our goal is to minimize type II error while controlling type I error under a target level, then we should operate under NP paradigm. Its results are shown above in Figure 7. We observe that type I error is well-controlled under α across all combinations of resampling techniques with classification models. Finally, we summarize the optimal combinations for NP paradigm in the table below.
When working with imbalanced data, it is critical to start with asking yourself which error is more costly to make in your problem: type I or type II? This question will help you to determine the appropriate classification paradigm to operate under. Once you identify the appropriate paradigm, select the optimal combination of resampling technique and classification model from the given charts here to apply on your dataset. Your selection should be based on the evaluation metric that you want to use to assess your classifier performance. There are several evaluation metrics available; choose the one that focuses on the error you find to be relatively costly. Nonetheless, please keep in mind that we have merely considered a selective list of resampling techniques and three classification models in our case study. There are many other combinations that are worthwhile to explore. Additionally note that in our case study, we randomly selected a subsample of the non-fraud class to construct our classifiers, which might be unrepresentative for our problem.
Applying classification paradigms and resampling techniques: We apply logistic regression as a mere example.
1. Classical Classification Paradigm
## Training setdata_trainxy <- data.frame(train_x,train_y)## Testing setdata_testxy <- data.frame(test_x,test_y) ## Fit the model on the training setmodel_cc = glm(train_y~.,family = “binomial”,data=data_trainxy)## Generate predictions on the testing setprep_cc = predict(model_cc,test_x)p_cc = 1/(1+exp(-prep_cc))## probability of Y=1 given X=x under testing set## Calculate the overall classification error (risk) on the testing set; set threshold=0.5threshold <- 0.5pred_class <- ifelse(p_cc>threshold,”1",”0")risk <- 1-mean(pred_class==test_y)
2. Cost-Sensitive Learning Paradigm (postprocessing method)
## Fit the model on the training setmodel_cs = glm(train_y~.,family = “binomial”,data=data_trainxy)prep_cs = predict(model_cs,test_x)p_cs = 1/(1+exp(-prep_cs)) ## the probability of Y=1 given X=x under the testing set## Generate predictions on the testing setthreshold_cs = IR/(IR+1) ## IR is the imbalance ratiopred_cs = ifelse(p_cs>threshold_cs,”1",”0")## Type I error on the testing setind0 = which(test_y == 0)typeI = mean(pred_cs[ind0] != test_y[ind0])## Type II error on the testing setind1 = which(test_y == 1)typeII = mean(pred_cs[ind1] != test_y[ind1]) ##Empirical misclassification cost on the testing setm0 = length(which(test_y==0)) # samples size of Class 0 in test datam1 = length(which(test_y==1)) # sample size of Class 1 in test datacost = IR*(m0/(m0+m1))*typeI+1*(m1/(m0+m1))*typeII
3. Neyman-Pearson Paradigm (NP umbrella algorithm)It is worthy to note that when constructing NP classifier, we can only apply resampling techniques onto a partial sample of the training set. It is because the algorithm behind it (NP umbrella) splits class 0 of the training set into two parts: one to fit the model and the other to select the threshold. The details of NP umbrella algorithm can be found in the following paper “Neyman-Pearson classification algorithms and NP receiver operating characteristics.”
## Install library “nproc” packageinstall.packages(“nproc”, repos = “http://cran.us.r-project.org")library(nproc)## Fit the model on the training setfit = npc(train_x, train_y, method = “logistic”)## Generate predictions on the testing setpred_np = predict(fit, test_x)## Compute type I error on the testing setind0 = which(test_y == 0)typeI = mean(pred_np[ind0] != test_y[ind0])## Compute type II error on the testing setind1 = which(test_y == 1)typeII = mean(pred_np[ind1] != test_y[ind1])
1. Random undersampling (“Under”)
id0_train = which(train_y==0) ## index of class 0 in the training setid1_train = which(train_y==1) ## index of class 1 in the training setn0_train = length(id0_train) ## sample size of class 0 in the training set## Randomly sample observations without replacement from class 1indexMaj = sample(id1_train,n0_train)## Obtain new training setnewtrain_under = data.frame(rbind(train_x[id0_train,],train_x[indexMaj,]), y=c(rep(0,n0_train),rep(1,n0_train)))
2. Oversampling (SMOTE)
## Install library “smote” packageinstall.packages(“smote”)library(smote)## Use SMOTE function to construct a new training setgeneData = SMOTE(train_x,train_y)## Obtain the new training setnewtrain_smote = geneData$data
3. Hybrid:Here it combines “Under” and SMOTE resampling techniques with the final training set consisting of the below:
## Calculate the new sample size of the new training setn0_train = length(id0_train) ## sample size of class 0 in the training setn1_train = length(id1_train) ## sample size of class 0 in the training setn_h = floor(n1_train/n0_train)*n0_train## Use SMOTE function to generate new data for class 0M = n_h/n0_train-1geneData_h = SMOTE(train_x,train_y,dup_size = M)newtrain_hybrid0 = geneData_h$data[which(geneData_h$data$class==”0"),]d=dim(newtrain_hybrid0)[2]## Randomly sample observations without replacement from class 1indexMaj_h = sample(id1_train, n_h)## Obtain the new training set with hybrid methodnewtrain_hybrid =data.frame(rbind(newtrain_hybrid0[1:d,],train_x[indexMaj_hybrid,]),y=c(rep(0,n_h),rep(1,n_h))) #the final training set consisting of n_h class 0 and n_h class 1 observations
Finally, please note that ROC-AUC and PR-AUC can be calculated by the functions of “roc.curve” and “pr.curve” respectively in “PRROC” package. For other evaluation metrics, you can simply use the formulas included here to compute them.
Many thanks to Professor Xin Tong, PhD for his kind and continuous support!