4 Unique Approaches To Manage Imbalanced Classification Scenarios | by Kaushik Choudhury | Towards Data Science

Out of billions of financial transactions, only a few involve cheating and fraud. Out of millions of cars on roads, only a few break down in the middle of the highway and rest drive fine. If we pay close attention to our daily activities, then a few exception incidents can also be identified. The same skewed data is present in many data points where one or a few couples of classes covers the majority of the cases.
When we provide these imbalanced data points to machine learning algorithms, a few majority classes heavily influence at the cost of ignoring the minority class. Most business cases relate to predicting minority class events like fraud, detection of alignment etc. The prediction accuracy of the rare event of the machine learning model trained on imbalanced data is abysmally bad.
In this article, I will discuss four unique ways to handle the imbalance dataset and improve the prediction accuracy of the minority class. Also, we will learn why only considering classification metric scores like F1 score or accuracy can mislead about model prediction performance in the rare event forecast.
We will be using the make_classification method in Scitkit-Learn to generate the imbalanced dataset.
import pandas as pdfrom sklearn.model_selection import train_test_splitfrom sklearn.datasets import make_classificationfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.metrics import f1_score,accuracy_scorefrom sklearn.metrics import plot_confusion_matriximport matplotlib.pyplot as plt
We will be learning the five unique approaches to handle the imbalanced dataset of 5000 samples with one class comprising of 98% cases.
X, y = make_classification(n_samples=5000,weights=[0.02, 0.98],                           random_state=0,n_clusters_per_class=1)ycount=pd.DataFrame(y)print(ycount[0].value_counts())
Out of 5000 sample records, we have 4871 records for class 1 and 129 class 0 records. Lets us consider class 1 suggests normal transaction and class 0 as fraudulent transactions.
Sample dataset is split into two parts viz. training and test set. The training set is to train the machine learning model, and the test set is to check the prediction of the model.
We will train the model with 80% of the sample data set and the remaining 20% records which model has not seen before is reserved for the testing set.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,random_state=42,stratify=y)
To understand the effect of the imbalanced dataset even on the sophisticated algorithm like Random Forest Classifier let us first train the standard Random Forest Classifier directly with the imbalanced training set and without any weight parameters.
clf =RandomForestClassifier(max_depth=2,random_state=0).fit(X_train, y_train)print("F1 Score is ", f1_score(y_test,clf.predict(X_test)))print("Accuracy Score is ", accuracy_score(y_test,clf.predict(X_test)))
The F1 Score and accuracy score of the trained Random Forest Classifier Model on the test dataset is high. But only considering only these metric to judge the prediction performance of the model can be very misleading in case of the imbalanced dataset.
Deploying such a model only based on these two metrics and not understanding the areas in which the classification model is making an error can be quite costly.
A visual metric, like the confusion matrix, outshines other metrics in several ways. We get an instant view on the model performance in terms of classification areas the model excelled and the areas which require fine-tuning. Based on the business use case, we can judge quickly from the false positive, false negative, true positive and true negative counts whether the model is ready for deployment.
You can learn in-depth about the confusion matrix in the article Accuracy Visualisation: Supervised Machine Learning Classification Algorithms
fig=plot_confusion_matrix(clf, X_test, y_test)plt.show()
As expected, the majority class has completely influenced the model, and the trained model has predicted the classification of all records in test dataset as the majority class. Such misclassification prediction in case of rare fraud detection or uncommon malignant disease prediction is very detrimental.
Fortunately, Random Forrest Classifier has a parameter “class_weight” to specify the weights of each class in case of an imbalanced dataset.
In the sample dataset, class 1 is approx 38 times more prevalent than class 0. Hence, we will mention the “class weights” in such proportion for the algorithm to compensate during training.
weighted_clf = RandomForestClassifier(max_depth=2, random_state=0,class_weight={0:38,1:1}).fit(X_train, y_train)print("F1 Score for RandomForestClassifier with class_weight parameter is ", f1_score(y_test,weighted_clf.predict(X_test)))print("Accuracy  Score for RandomForestClassifier with class_weight parameter is ", accuracy_score(y_test,weighted_clf.predict(X_test)))
The F1 Score and accuracy score for Random Forest Classifier Model with class weigh compensated is also high, but we can ascertain the real performance by checking the confusion matrix.
We can see that the majority class has not completed overtaken the Random Forest Classifier Model with weight-adjusted.
fig=plot_confusion_matrix(weighted_clf, X_test, y_test)plt.show()
Out of the total 1000 records in the test dataset, it has misclassified only 14 records. Also, it has correctly classified 20 of the minority class sample records out of 26 minority class records in the test dataset.
We have learned the way to handle imbalanced dataset with class_weight parameter in Random Forrest Classifier and improve the prediction accuracy of the minority class.
Next, we will learn a different approach to manage the imbalance input training dataset with BalancedRandomForestClassifier in the imbalanced-learn library.
Install the imbalance-learn library with pip
pip install imbalanced-learn
In the below code, we trained the BalancedRandomForestClassifier with the training dataset and then checked the metrics scores on the testing dataset.
from imblearn.ensemble import BalancedRandomForestClassifierbrfc = BalancedRandomForestClassifier(n_estimators=500,random_state=0).fit(X_train,y_train)print("F1 Score for Balanced Random Forest Classifier is ", f1_score(y_test,brfc.predict(X_test)))print("Accuracy  Score for Balanced Random Forest Classifier is ", accuracy_score(y_test,brfc.predict(X_test)))
Like the previous two examples, it also indicates high F1 and accuracy score.
We can see in the confusion matrix that BalancedRandomForestClassifier handles the class weight internally quite well compare to RandomForestClassifier without weight_class parameter.
fig=plot_confusion_matrix(brfc, X_test, y_test)plt.show()
Out of 1000 test records, it predicted the classification of 968 records correctly. It also performed slightly better than Random Forrest Classifier with class_weight by correctly classifying 21 records out of 26 records in the minority class.
Next, we will use a completely different approach of oversampling to manage the minority class in the training dataset.
Next, we will use a completely different approach of oversampling to manage the minority class in the training dataset. The basic idea is to randomly generate examples in the minority class to have a more balanced dataset.
from imblearn.over_sampling import RandomOverSamplerros = RandomOverSampler(random_state=0)X_resampled, y_resampled = ros.fit_resample(X_train, y_train)print("Number of records for X_train is ", X_train.shape)print("Number of records for X_resampled oversampling is ",X_resampled.shape)
Earlier we divided the sample dataset of 5000 records into training and test dataset with 4000 and 1000 records respectively.
Training dataset fit on RandomOverSampler generated minority class record at random and resampled balanced training data has 7794 records.
Once the training dataset is artificially balanced, then we can train the standard Random Forest Classifier without “class_weight” parameter.
oclf = RandomForestClassifier(max_depth=2, random_state=0).fit(X_resampled, y_resampled)
We see that standard Random Forest classifier trained on artificially balanced training dataset with oversampling could predict pretty good.
Oversampling helped the Random Classifier to overcome the influence of majority classifier and predict the test data record classes with high accuracy.
Out of 1000 test records, it predicted the classification of 985 records correctly. It also performed nearly at par with BalancedRandomForestClassifier with classifying 20 records out of 26 records in the minority class.
Finally, we will learn about the undersampling strategy to handle the imbalanced dataset. It is a completely different approach than oversampling we learnt earlier. Randomly delete examples in the majority class. The key idea is to randomly delete the majority class records to have a more balanced dataset.
from imblearn.under_sampling import RandomUnderSamplerrus = RandomUnderSampler(random_state=0)X_resampled, y_resampled = rus.fit_resample(X_train, y_train)print("Number of records for X_train is ", X_train.shape)print("Number of records for X_resampled undersampling is ",X_resampled.shape)
A number for majority class records deleted at random to balance the training data set with 206 records from 4000 data records.
Once the training dataset is balanced, we can use it directly to train the model just like oversampling strategy discussed earlier.
uclf=RandomForestClassifier(max_depth=2,random_state=0).fit(X_resampled, y_resampled)
It seems undersampling strategy enables to predict the rare minority class event with similar accuracy like other strategies discussed in this article, but in comparison to other strategies, it performed pretty badly in of predicting the majority class. It predicted incorrectly 82 majority class records from the test dataset.
fig=plot_confusion_matrix(uclf, X_test, y_test)plt.show()
Key takeaways and my approach
Most of the machine learning classification algorithms expect balanced training dataset. It is vital to check whether the training dataset is imbalanced and take appropriate pre-processing actions before training the machine learning model with the data.
GIGO — Garbage In and Garbage Out: If we train a model with imbalanced data, then there is a high probability that the model will miss to predict the minority classes in production.
Data is very valuable. I do not prefer the undersampling strategy as it forces to prune the data related to the majority class. We saw that due to this even though the model could predict the minority class records nearly with the same accuracy as other strategies discussed in this article it performed miserably in predicting the majority class records.
I prefer the Random Forest Classifier with ‘class_weight’ parameter and BalancedRandomForestClassifier in imbalanced-learn library.
I will suggest you check the training sample performance with all the strategies discussed in this article before selecting any one of them for your project.
You can gain insight into the imbalanced dataset with exploratory data analysis. To know more about it read the article- 5 Advanced Visualisation for Exploratory data analysis (EDA)