Dealing with Imbalanced dataset. Techniques to handle imbalanced data | by Vaibhav Jayaswal | Towards Data Science

The imbalanced dataset in real-world problems is not so rare. In layman terms, an imbalanced dataset is a dataset where classes are distributed unequally. An imbalanced data can create problems in the classification task. Before delving into the handling of imbalanced data, we should know the issues that an imbalanced dataset can create.
We will take an example of a credit card fraud detection problem to understand an imbalanced dataset and how to handle it in a better way.
EXAMPLE — CREDIT CARD FRAUD DETECTION
The dataset of credit card fraud detection is taken from Kaggle. The dataset contains transactions that occurred in two days, where
Total transactions in the data = 284,807
The data has two classes: 0 and 1 represents legal transactions and fraud transactions, respectively. Plotting the distribution of the classes will give us insights about the imbalance if present.
The total legit transactions are 284315 out of 284807, which is 99.83%. The fraud transactions are only 492 in the whole dataset (0.17%). An imbalanced dataset can occur in other scenarios such as cancer detection where large amounts of tested people are negative, and only a few people have cancer.
If we are using accuracy as a performance metric, it can create a huge problem. Let’s say our model predicts each transaction as legal (dumb model). Using an accuracy metric on the credit card dataset will give 99.83% accuracy, which is excellent. IS IT A GOOD RESULT? NO
For an imbalanced dataset, other performance metrics should be used, such as the Precision-Recall AUC score, F1 score, etc.. Moreover, the model will be biased towards the majority class. Since most machine learning techniques are designed to work well with a balanced dataset, we must create balanced data out of an imbalanced dataset. HOW TO DO THAT?
NOTE: Before changing the dataset, we must split the dataset into training and testing because the change is only for the training purpose.
>>>from sklearn.model_selection import train_test_split>>>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.33, random_state = 2, shuffle = True, stratify = y)
The shape of the training dataset:0: 190430 transactions1: 330 transactions
We will discuss three methods in this article for creating a balanced dataset from imbalanced data:
Undersampling
Oversampling
Creating synthetic data
Undersampling resamples the majority class points in the data to make them equal to the minority class points. We will be creating a new dataset out of the original dataset using undersampling. We will randomly sample the legal transactions (190490 in number) in the credit card fraud detection to make data points of class 0 equal to 330 (minority class).
>>>from imblearn.under_sampling import RandomUnderSampler>>>rus = RandomUnderSampler(random_state=0)>>>X_resampled_under, y_resampled_under =rus.fit_resample(X_train, y_train)
The major disadvantage of undersampling is that we do not use a significant chunk of the data, which contains some information. In our example, we have removed 190160 (190490–330) data points. Therefore, we are losing information, and as a result, we will not get significant results.
We can avoid that by using a technique called oversampling instead of undersampling.
Oversampling refers to the resampling of the minority class points to equal the total number of majority points. Repetition of the minority class points is one such type of oversampling technique.
Apart from repetition, we can provide the class weights to both the class. Providing the large weights to the minority class will give the same result as from that of repetition.
The problem with repeating the data is that it does not provide any extra information. One way to increase the information about the data is by creating synthetic data points. One such technique is the SMOTE (Synthetic Minority Oversampling technique). As the name suggests, SMOTE is an oversampling technique. In layman terms, SMOTE will create synthetic data points for the minority class. It creates new instances between the points of the minority class.
We can apply SMOTE on the training dataset using the imblearn library.
>>>from imblearn.over_sampling import SMOTE>>>sm = SMOTE(random_state = 42)>>>X_train_new, y_train_new = sm.fit_sample(X_train, y_train)
The imbalanced dataset is extremely common when handling real-world scenarios. A machine learning model is not robust if it uses an imbalanced dataset for training purposes. Therefore, a balanced dataset is preferred for training machine learning models. Techniques such as undersampling, oversampling, and SMOTE can be used to create balanced data.
Thanks for reading!