4 Unique Approaches To Manage Imbalanced Classification Scenarios
By: Kaushik Choudhury
Tags: Programming, Scikit Learn, Machine Learning, Data Science, Python

We will be using the make_classification method in Scitkit-Learn to generate the imbalanced dataset.

import pandas as pd

from sklearn.model_selection import train_test_split

from sklearn.datasets import make_classification

from sklearn.ensemble import RandomForestClassifier

from sklearn.metrics import f1_score,accuracy_score

from sklearn.metrics import plot_confusion_matrix

import matplotlib.pyplot as plt

We will be learning the five unique approaches to handle the imbalanced dataset of 5000 samples with one class comprising of 98% cases.

X, y = make_classification(n_samples=5000,weights=[0.02, 0.98],

random_state=0,n_clusters_per_class=1) ycount=pd.DataFrame(y)

print(ycount[0].value_counts())

The proportion of majority and minority class in-sample data set ( output of the above code)

Out of 5000 sample records, we have 4871 records for class 1 and 129 class 0 records. Lets us consider class 1 suggests normal transaction and class 0 as fraudulent transactions.

Sample dataset is split into two parts viz. training and test set. The training set is to train the machine learning model, and the test set is to check the prediction of the model.

We will train the model with 80% of the sample data set and the remaining 20% records which model has not seen before is reserved for the testing set.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,random_state=42,stratify=y)

To understand the effect of the imbalanced dataset even on the sophisticated algorithm like Random Forest Classifier let us first train the standard Random Forest Classifier directly with the imbalanced training set and without any weight parameters.

clf =RandomForestClassifier(max_depth=2,random_state=0).fit(X_train, y_train) print("F1 Score is ", f1_score(y_test,clf.predict(X_test)))

print("Accuracy Score is ", accuracy_score(y_test,clf.predict(X_test)))

The F1 Score and accuracy score of the trained Random Forest Classifier Model on the test dataset is high. But only considering only these metric to judge the prediction performance of the model can be very misleading in case of the imbalanced dataset.

F1 and Accuracy score of Random Forest Classifier trained on the imbalanced dataset ( output of the above code)

Deploying such a model only based on these two metrics and not understanding the areas in which the classification model is making an error can be quite costly.

A visual metric, like the confusion matrix, outshines other metrics in several ways. We get an instant view on the model performance in terms of classification areas the model excelled and the areas which require fine-tuning. Based on the business use case, we can judge quickly from the false positive, false negative, true positive and true negative counts whether the model is ready for deployment.

You can learn in-depth about the confusion matrix in the article Accuracy Visualisation: Supervised Machine Learning Classification Algorithms

fig=plot_confusion_matrix(clf, X_test, y_test)

plt.show()

As expected, the majority class has completely influenced the model, and the trained model has predicted the classification of all records in test dataset as the majority class. Such misclassification prediction in case of rare fraud detection or uncommon malignant disease prediction is very detrimental.

Confusion Matrix of test dataset prediction by Random Forest Classifier trained on the imbalanced dataset (output of the above code)

Fortunately, Random Forrest Classifier has a parameter “class_weight” to specify the weights of each class in case of an imbalanced dataset.

In the sample dataset, class 1 is approx 38 times more prevalent than class 0. Hence, we will mention the “class weights” in such proportion for the algorithm to compensate during training.

weighted_clf = RandomForestClassifier(max_depth=2, random_state=0,class_weight={0:38,1:1}).fit(X_train, y_train) print("F1 Score for RandomForestClassifier with class_weight parameter is ", f1_score(y_test,weighted_clf.predict(X_test))) print("Accuracy Score for RandomForestClassifier with class_weight parameter is ", accuracy_score(y_test,weighted_clf.predict(X_test)))

The F1 Score and accuracy score for Random Forest Classifier Model with class weigh compensated is also high, but we can ascertain the real performance by checking the confusion matrix.

F1 and Accuracy score of Random Forest Classifier trained on the class optimised dataset with class_weight parameter ( output of the above code)

We can see that the majority class has not completed overtaken the Random Forest Classifier Model with weight-adjusted.

fig=plot_confusion_matrix(weighted_clf, X_test, y_test)

plt.show()

Out of the total 1000 records in the test dataset, it has misclassified only 14 records. Also, it has correctly classified 20 of the minority class sample records out of 26 minority class records in the test dataset.

Confusion Matrix of test dataset prediction by Random Forest Classifier trained on the weight-optimised dataset (output of the above code)

We have learned the way to handle imbalanced dataset with class_weight parameter in Random Forrest Classifier and improve the prediction accuracy of the minority class.

Next, we will learn a different approach to manage the imbalance input training dataset with BalancedRandomForestClassifier in the imbalanced-learn library.

Install the imbalance-learn library with pip

pip install imbalanced-learn

In the below code, we trained the BalancedRandomForestClassifier with the training dataset and then checked the metrics scores on the testing dataset.

from imblearn.ensemble import BalancedRandomForestClassifier brfc = BalancedRandomForestClassifier(n_estimators=500,

random_state=0).fit(X_train,y_train) print("F1 Score for Balanced Random Forest Classifier is ", f1_score(y_test,brfc.predict(X_test))) print("Accuracy Score for Balanced Random Forest Classifier is ", accuracy_score(y_test,brfc.predict(X_test)))

Like the previous two examples, it also indicates high F1 and accuracy score.

F1 and Accuracy score of Balanced Random Forest Classifier trained ( output of the above code)

We can see in the confusion matrix that BalancedRandomForestClassifier handles the class weight internally quite well compare to RandomForestClassifier without weight_class parameter.

fig=plot_confusion_matrix(brfc, X_test, y_test)

plt.show()

Out of 1000 test records, it predicted the classification of 968 records correctly. It also performed slightly better than Random Forrest Classifier with class_weight by correctly classifying 21 records out of 26 records in the minority class.

Confusion Matrix of test dataset prediction by Balanced Random Forest Classifier (output of the above code)

Next, we will use a completely different approach of oversampling to manage the minority class in the training dataset.

Next, we will use a completely different approach of oversampling to manage the minority class in the training dataset. The basic idea is to randomly generate examples in the minority class to have a more balanced dataset.

from imblearn.over_sampling import RandomOverSampler

ros = RandomOverSampler(random_state=0)

X_resampled, y_resampled = ros.fit_resample(X_train, y_train)

print("Number of records for X_train is ", X_train.shape)

print("Number of records for X_resampled oversampling is ",X_resampled.shape)

Earlier we divided the sample dataset of 5000 records into training and test dataset with 4000 and 1000 records respectively.

Training dataset fit on RandomOverSampler generated minority class record at random and resampled balanced training data has 7794 records.

Training dataset count with oversampled strategy- generated minority class record at random to balance the training dataset (Output of the above code)

Once the training dataset is artificially balanced, then we can train the standard Random Forest Classifier without “class_weight” parameter.

oclf = RandomForestClassifier(max_depth=2, random_state=0).fit(X_resampled, y_resampled)

We see that standard Random Forest classifier trained on artificially balanced training dataset with oversampling could predict pretty good.

Oversampling helped the Random Classifier to overcome the influence of majority classifier and predict the test data record classes with high accuracy.

Confusion Matrix of test dataset prediction by Random Forest Classifier trained on the oversampled dataset (output of the above code)

Out of 1000 test records, it predicted the classification of 985 records correctly. It also performed nearly at par with BalancedRandomForestClassifier with classifying 20 records out of 26 records in the minority class.

Finally, we will learn about the undersampling strategy to handle the imbalanced dataset. It is a completely different approach than oversampling we learnt earlier. Randomly delete examples in the majority class. The key idea is to randomly delete the majority class records to have a more balanced dataset.

from imblearn.under_sampling import RandomUnderSampler rus = RandomUnderSampler(random_state=0)

X_resampled, y_resampled = rus.fit_resample(X_train, y_train) print("Number of records for X_train is ", X_train.shape)

print("Number of records for X_resampled undersampling is ",X_resampled.shape)

A number for majority class records deleted at random to balance the training data set with 206 records from 4000 data records.

Training dataset count with under-sampling strategy. Deleted majority class record at random to balance the training dataset (Output of the above code)

Once the training dataset is balanced, we can use it directly to train the model just like oversampling strategy discussed earlier.

uclf=RandomForestClassifier(max_depth=2,

random_state=0).fit(X_resampled, y_resampled)

It seems undersampling strategy enables to predict the rare minority class event with similar accuracy like other strategies discussed in this article, but in comparison to other strategies, it performed pretty badly in of predicting the majority class. It predicted incorrectly 82 majority class records from the test dataset.

fig=plot_confusion_matrix(uclf, X_test, y_test)

plt.show()

Confusion Matrix of test dataset prediction by Random Forest Classifier trained on the undersampling dataset (output of the above code)

Key takeaways and my approach

Most of the machine learning classification algorithms expect balanced training dataset. It is vital to check whether the training dataset is imbalanced and take appropriate pre-processing actions before training the machine learning model with the data.

GIGO — Garbage In and Garbage Out: If we train a model with imbalanced data, then there is a high probability that the model will miss to predict the minority classes in production.

Data is very valuable. I do not prefer the undersampling strategy as it forces to prune the data related to the majority class. We saw that due to this even though the model could predict the minority class records nearly with the same accuracy as other strategies discussed in this article it performed miserably in predicting the majority class records.

I prefer the Random Forest Classifier with ‘class_weight’ parameter and BalancedRandomForestClassifier in imbalanced-learn library.

I will suggest you check the training sample performance with all the strategies discussed in this article before selecting any one of them for your project.

You can gain insight into the imbalanced dataset with exploratory data analysis. To know more about it read the article- 5 Advanced Visualisation for Exploratory data analysis (EDA)