Techniques for Handling Imbalanced Classification Datasets
By: Renu Khandelwal
Tags: Imbalanced Data, Machine Learning, Python, Imblearn, Smote

Techniques for Handling Imbalanced Classification Datasets

Learn the common techniques to handle Imbalanced classification datasets for structured data

In this article, you will learn about the Imbalanced dataset and issues that arise when a classification dataset is Imbalanced. Understand the common techniques like Over Sampling, Under Sampling, generate synthetic data to handle imbalanced datasets, and finally, apply all the concepts to an imbalanced dataset.

Photo by Azzedine Rouichi on Unsplash

An imbalanced dataset is one in which one class has disproportionate observations compared to the other classes. Each class in the dataset does not have an equal representation and imbalance causes the skewed class distribution.

You have to run a classification algorithm to distinguish between a benign tumor and a cancerous tumor. There 20,000 observations with benign tumors and just 100 observations related to cancerous tumors; this causes the dataset to be imbalanced.

The imbalanced dataset's impact is visible when the prediction for an observation is benign when it was actually a cancerous tumor resulting in a false negative.

Accuracy is the most common performance metric for classification but avoids using Accuracy as a performance metric as it can be misleading, especially in the case of an imbalanced dataset. When we have a disproportionate representation of classes in a dataset, a model is likely to predict the majority class's value for all predictions and achieve a high classification accuracy, also referred to as Accuracy Paradox.

To ensure a model works great in the real world, false positives and false negatives need to be kept to a minimum. Precision helps calculate the costs of false positives, and Recall helps understand the cost of false negatives. Hence precision, recall, and F1-score are better measures of a model’s performance.

Image by Author

Common techniques to handle imbalanced datasets

Cost-Sensitive Training takes the misclassification costs of the minority class into consideration by penalizing the machine learning algorithms. The idea is to minimize the cost of classification. Under-sampling or down-sampling the majority class where the observations of the majority class are randomly removed to reduce its influence on the machine learning algorithm Over-sampling or up-sampling, the minority class where the observations for the minority class is randomly duplicated to increase the influence on the machine learning algorithm Creating synthetic data for the minority class using SMOTE(Synthetic Minority over-sampling Technique). SMOTE takes random samples from the minority class, finds its nearest k neighbors, and then selects a point between the randomly selected data point and its nearest k neighbors to generate synthetic data.

SMOTE using nearest k-neighbors to generate synthetic data(Image by Author)

You will take an imbalanced dataset for insurance cross-sell to build logistics regression to predict whether an existing customer would be interested or not in vehicle insurance.

Importing required libraries and reading the data into a data frame

import numpy as np

import pandas as pd

from sklearn.preprocessing import StandardScaler

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LogisticRegression

from sklearn.metrics import confusion_matrix

from sklearn.metrics import plot_confusion_matrix

from sklearn.metrics import precision_score, recall_score, accuracy_score, f1_score

import matplotlib.pyplot as plt

import seaborn as sns

%matplotlib inline #Reading the data from the csv file into a dataframe

data= pd.read_csv(r'\Insurance_data\train.csv')

Displaying the dataset information

data.info()

Convert categorical variables to Indicator columns

We have a few categorical variables like gender and vehicle damage that needs to be converted to dummy or indicator variables. Here we remove the first level to get k-1 dummies out of k categorical levels by setting the drop_first parameter to True.

data_2 = pd.get_dummies(data,drop_first=True)

Viewing the data

Creating features and the target variable

X= data_2.iloc[:,[1,2,3,4,5,6,7,9,10,11,12]]

y=data_2.iloc[:,8]

Viewing the target variable distribution

Checking on the target variable distribution to check if the dataset is balanced or not.

y.value_counts().plot(kind='bar',figsize=(3,3),title="Insurance sell")

plt.xlabel("Insurance Response")

plt.ylabel("Frequency")

y_pos=50000

x_pos=0

for i, v in enumerate(data_count):

plt.text(i+x_pos, y_pos, v, horizontalalignment='center', verticalalignment='center', fontsize=20)

plt.show()

The dataset is not balanced as we have more observations for the customer not interested in buying vehicle insurance than customers interested in buying vehicle insurance.

Applying Standardscalar to standardize the features

# define standard scaler

scaler = StandardScaler()

X= pd.DataFrame(scaler.fit_transform(X))

Splitting dataset into training and test

X_train, X_test, y_train, y_test= train_test_split(X,y,test_size=0.2)

Accuracy Paradox for Imbalanced dataset

Running logistics regression on the imbalanced training dataset showing the accuracy, precision, recall, and F1-score.

clf = LogisticRegression(random_state=0).fit(X_train, y_train)

y_pred=clf.predict(X_test)

Plotting the Confusion Matrix

plot_confusion_matrix(clf,X_test, y_test)

Confusion Matrix using logistics regression on an imbalanced dataset

Checking accuracy, precision, recall, and F1-score on the imbalanced test dataset.

print("Accuracy",accuracy_score(y_test, y_pred))

print("Precision", precision_score(y_test, y_pred))

print("Recall",recall_score(y_test, y_pred))

print("F1 score",f1_score(y_test, y_pred))

You can see that we have relatively good accuracy, the recall and F1-score are very bad. This demonstrates the Accuracy paradox where accuracy is not a good measure of the model's performance. It is a good idea to check the precision, recall, and F1-score.

Applying Cost-Sensitive Training

Applying class_weight=”balanced” will automatically adjust weights based on the class frequency. Class weights are inversely proportional to class frequencies in the input data.

class weights = Total no. of records in the dataset/( Total no. of classes * No. of samples in the class)

clf_cw = LogisticRegression(random_state=0, class_weight='balanced').fit(X_train, y_train) y_pred= clf_cw.predict(X_test) print("Accuracy",accuracy_score(y_test, y_pred))

print("Precision", precision_score(y_test, y_pred))

print("Recall",recall_score(y_test, y_pred))

print("F1 score",f1_score(y_test, y_pred))

You can see that the accuracy has dropped, but we have a better F1 score.

Displaying confusion matrix after applying the class_weights

plot_confusion_matrix(clf_cw, X_test, y_test)

Confusion Matrix applying class_weights to an imbalanced dataset

Install imblearn library for handling imbalanced dataset using different techniques

pip install imblearn

Applying Random Under Sampling

Here you will use a class to perform random under-sampling, which down-samples the majority class(es) by randomly picking samples with or without replacement.

Comparing the number of observation between the original dataset and the training dataset generated using Random Under Sampling

Under Sampling or downsampling majority class

from imblearn.under_sampling import RandomUnderSampler # create the train and test dataset

X_train, X_test, y_train, y_test= train_test_split(X, y, test_size=0.2) # Create an instance of RandomUnderSampler, fit the training data #and apply Logistics regression

rus= RandomUnderSampler()

X_train, y_train = rus.fit_resample(X_train, y_train)

clf_rus= LogisticRegression(random_state=0).fit(X_train, y_train) # predict the test data

y_pred= clf_rus.predict(X_test) # print the model performance metrics

print("Accurcay",accuracy_score(y_test, y_pred))

print("Precision", precision_score(y_test, y_pred))

print("Recall",recall_score(y_test, y_pred))

print("F1 score",f1_score(y_test, y_pred))

plot_confusion_matrix(clf_rus, X_test, y_test)

Performance metrics using Random Under Sampler

Confusion Metrics using Random Under Sampler

Applying Random Over Sampling

Random Oversampling over-samples the minority class(es) by picking samples at random with replacement. You can specify the sampling strategy as a parameter to resample the data set.

Comparing the number of observations between the original dataset and the training dataset generated using Random OverSampling.

Over Sampling or upsampling minority class

from imblearn.over_sampling import RandomOverSampler # create the train and test dataset

X_train, X_test, y_train, y_test= train_test_split(X, y, test_size=0.2) # Create an instance of RandomOverSampler, fit the training data #and apply Logistics regression

ros= RandomOverSampler(sampling_strategy='auto')

X_train, y_train = ros.fit_resample(X_train, y_train)

clf_ros= LogisticRegression(random_state=0).fit(X_train, y_train) # predict the test data

y_pred= clf_ros.predict(X_test) # print the model performance metrics

print("Accurcay",accuracy_score(y_test, y_pred))

print("Precision", precision_score(y_test, y_pred))

print("Recall",recall_score(y_test, y_pred))

print("F1 score",f1_score(y_test, y_pred))

plot_confusion_matrix(clf_ros, X_test, y_test)

Performance metrics using Random Over Sampler

Applying Synthetic Minority Over-sampling Technique(SMOTE)

SMOTE performs over-sampling by taking random samples from the minority class, finding its nearest k neighbors, which is a parameter that can be specified. Selects a point between the randomly selected data point and its nearest k neighbors to generate synthetic data.

Comparing the number of observations between the original dataset and the training dataset generated using SMOTE.

Applying SMOTE to generate synthetic data

from imblearn.over_sampling import SMOTE # create the train and test dataset

X_train, X_test, y_train, y_test= train_test_split(X, y, test_size=0.2) ##Create an instance of SMOTE, fit the training data and apply #Logistics regression

sm = SMOTE(random_state=27, sampling_strategy='minority', k_neighbors=5)

X_train, y_train = sm.fit_resample(X_train, y_train)

clf_sm = LogisticRegression(random_state=0).fit(X_train, y_train) # predict the test data

y_pred= clf_sm.predict(X_test) # print the model performance metrics

print("Accurcay",accuracy_score(y_test, y_pred))

print("Precision", precision_score(y_test, y_pred))

print("Recall",recall_score(y_test, y_pred))

print("F1 score",f1_score(y_test, y_pred))

plot_confusion_matrix(clf_sm, X_test, y_test)

Performance metrics using SMOTE

Conclusion:

A dataset in which one class has substantially more observations than the other class(es) creates an imbalance dataset skewing the test results. There are different techniques like oversampling the minority class, under-sampling or downsampling the majority class, cost-sensitive techniques, or creating synthetic data using SMOTE.

Comparison of the number of observations of classes applying different techniques to handle imbalanced datasets(Image by Author)