How to Handle Imbalance Data and Small Training Sets in ML | by Ege Hosgungor | Towards Data Science

Imbalanced data refers to where the number of observations per class is not equally distributed and often there is a major class that has a much larger percentage of the dataset and minor classes which doesn’t have enough examples.
Small Training Sets also suffer from not having enough examples. Both of the problems are very common in real-world applications but luckily there are several ways to overcome this problem. This article will walk through many different techniques and perspectives to combat Imbalance data. In particular, you will learn about:
Sampling Techniques (Upsampling & Downsampling)
Weighted Loss
Data Augmentation Techniques
Transfer Learning
Imbalanced data is a common problem in data science. From image classification to fraud detection or medical diagnosis, data scientists face imbalanced datasets. Having an imbalanced dataset decreases the sensitivity of the model towards minority classes. Lets put this on with simple maths:
Imagine you have 10000 lung X-Ray images and only 100 of them are diagnosed with Pneumonia which is an infectious disease that inflames the air sacs in one or both lungs and fills them with liquid. If you train a model that predicts every example as healthy you will get 99% accuracy. Wow, how awesome is that? Wrong, you just killed many people with your model.
Usually, we look at accuracy to measure our model’s performance but imbalanced data can mislead us like the example above. This article is focused on how to handle these kinds of situations and I am not going into details of how to measure your model but bear in mind to change the measurement metrics from Accuracy to Precision, Recall, F1 Score. But even they are not actually that great. I would recommend going for:AUC Curves, Average Precision while dealing with imbalance data. Here is an amazing github repository which goes into very details of these metrics by talking about a particular imbalance dataset example: credit card frauds.
Let’s say, we have an imbalanced dataset and we do not have access to more data. So we should create our own data. The following chapter will present techniques on a high abstract level and I also added some great references for implementing each of them in Python.
You can balance your data by resampling them. The followings are two different techniques for resampling:
Upsampling (increase your minority class)
Downsample (decrease your majority class)
For both of these, we will use the Sklearn Resample function. Let’s import the libraries and define our data as df:
# Importing the librariesimport numpy as npimport pandas as pdfrom sklearn.utils import resample# Importing the dataset# Read datasetdf= pd.read_csv('data.csv')
In this example, we have a binary classification problem where the majority is represented as 1 and the minority class is represented as 0. Let’s separate them:
# Separate majority and minority classesdf_majority = df[df.iloc[:,4608]==1]df_minority = df[df.iloc[:,4608]==0]
We can downsample the majority class, upsample the minority class or we can do both. You should try them to find which one is the most suitable for you. One other hyperparameter that you can tune is n_samples which is the number of samples that you resample.
# Separate majority and minority classesdf_majority = df[df.iloc[:,4608]==1]df_minority = df[df.iloc[:,4608]==0] # Downsample majority classdf_majority_downsampled = resample(df_majority,                                  replace=False,                                     n_samples=1000)#Upsample minority classdf_minority_upsampled = resample(df_minority,                                  replace=True,                                      n_samples=1000)# Combine minority class with downsampled majority classdf_up_down_sampled = pd.concat([df_majority_downsampled, df_minority_upsampled])
Lastly display new class counts with:
df_downsampled.prediction.value_counts()
Congratulations, you brought balance to the power. 🎉
Edit:An interview question which I got asked lately: How would you make sure that your upsampled & downsampled data comes from a good distribution/ how would you measure the performance of these methods ?
One easy way to measure the performance of these methods are getting the test results for your new model which is trained with this additional data. Most of the time while upsampling you might first see an increase on the performance and then plateau or even decrease afterwards. I would pick the least number of upsampled data with the top performance.
But sometimes resampling doesn’t enough on its own. The article continues with more advanced techniques.
One other way to avoid having class imbalance is to weight the losses differently. To choose the weights, you first need to calculate the class frequencies.
# Count up the number of instances of each class (drop non-class columns from the counts)class_counts = df_classes.sum()#print the class frequenciesfor column in class_counts.keys():    print(f"The class {column} has {df_classes[column].sum()} samples")
Changing the weights in the loss function allows data scientists to balance the contribution of each class. One way of doing this is by multiplying each example from each class by a class-specific weight factor so that the overall contribution of each class is the same.
Thanks to the Sklearn, there is a built-in parameter called class_weight in most of the ML algorithms which helps you to balance the contribution of each class. For example, in the Sklearn’s RandomForestClassifier you have options as balanced, balanced_subsample or you can give weights for each class manually as a dictionary.
#Fitting RandomForestClassifierfrom sklearn.ensemble import RandomForestClassifiermodel = RandomForestClassifier(n_estimators=1000, class_weight={0:0.10, 1:0.90})
Amazing🥳, now you learned a very common technique that you can apply almost every kind of data with every ML algorithms. Let’s dive in even more advanced techniques.
From now onwards, the techniques that I’m going to mention are suitable mostly for Computer Vision and images. There are ways to generate data when you are working with images.
Data Augmentation is known as a very powerful technique used to artificially create variations in existing images to expand an existing image data set. This creates new and different images from the existing image data set that represents a comprehensive set of possible images. Here are several ways that you can do to increase your dataset.
Flipping
Zoom
Rotating
Shearing
Cropping
And lastly generating new images with Generative Adversarial Networks.
Note: Sometimes using these techniques can harm your model more than helping it. Let’s assume the same Lung X-Ray dataset case where we have many different X-rays. If you flip the images you will have a heart in the right which can be seen as a cist. Again you killed people =( So be careful about using these augmentation techniques.
You can learn more about this here
If you don’t have enough data to generalize your model you can use pre-trained models and tune them for your own task. This is called Transfer Learning and it is very common in the Computer Vision field. Since the training, huge models take so much time and resources, and people are getting pre-trained models like DenseNet and use them. The approach is as follows:
Find a model which suits your requirements
Re-use the model. This may involve using all or parts of the model or the first layers of the neural network. Since the complexity of features increases with the number of layers, the first layers always have more generic features that can be used successfully.
Fine-tuning: By going deeper, the neural network gets specialized on the task so you may want to tune the last couple of layers for your own task which is called Fine-tuning.
You can learn more from here
We learned 4 different techniques that you can apply to your imbalanced data / small training set. Now you know how to increase your model’s performance with:
Using Sampling Techniques
changing the Class Weights of Loss Function
Using Data Augmentation Techniques
and using Transfer Learning
The last two of them are more specific to Computer Vision. I hope this article helps you to understand how to overcome imbalance data problems with different techniques. Congratulations on finishing this article and thanks for reading all along.