Imbalanced Data,  What Can You Do? | Towards Data Science

When we speak of imbalanced data, what we mean is that at least one class is underrepresented. For example, when considering the problem of building a classifier, let’s call it the Idealisstic-Voter. We give it the task of identifying politicians that the American public finds trustworthy. When looking at the dataset, we realize that almost every sample is hugely unpopular.
In classifying terminology we call the classes with very few samples the minority-classes and the ones with many the majority-classes.
Let’s assume there are 100 politicians, and only one of them is considered trustworthy by the broad public; let’s call him Abraham. We could now quickly build an Idealisstic-Voter. That achieves a Precision of 99% by simply saying “all politicians are liars,” i.e. classifying all politicians as untrustworthy. The loss will be low, and we can make an excellent presentation about how great our Idealisstic-Voter scores on the most common metrics.
PRECISION = TP / (TP+FP) = 99%ACCURACY = TP / (TP + FN) = 99 %F1 = (2*PRECISION*RECALL)/(PRECISION+RECALL) = 99.5 %
But that is not the goal! What we want is to split the good from the bad, and we really did not split anything. So how can we detect that we have a problem? That we did injustice to Abraham, and we all know he does not like injustices ...
Whenever you are working with imbalanced data, make it a habit to also look at the balanced metrics. They do the same as the ones you are familiar with, but additionally, they take the average over all classes.
BALANCED_PRECISION=(P_TRUSTWORTHY + P_UNTRUSTWORTHY)/2=50%
Another metric that works very nicely in this context is the False Positive Rate in comparison with the True Positive Rate. In the multilabel case, you will mostly opt for some macro-averaged score.
We realized that we have a problem, how can we solve it?
Probably the simplest way you can use to make the Idealisstic-Voter solve the problem by himself is by presenting him with more reasonable options. This would mean either collecting more data or using the lovely Upsampling technique.
This technique works such that we clone our beloved Abraham, meaning that we duplicate the members of the minority class. Once we balanced the dataset to a sufficient degree, we will notice that our model has actually to learn something to attain a good score!
The opposite of Upsampling is Downsampling, aka Undersampling. This technique is great if you have more than enough data to do your job (Which never happened to me, but I assume the big data collectors of this world may sometimes be in this position).
In this case, you can eliminate some of the instances of the majority class. You don’t have to go 50–50, but enough to make the Idealisstic-Voter motivated to learn something. The same holds for all of the techniques we are discussing today.
Ok, now we are getting into the cool things. How about we generate data out of thin air? How about we add some Abrahams with cool techno glasses? He’d surely be just as trustworthy and additionally win the Youth Vote!
Let’s say our dataset has the feature age. We can clearly see that the only trustworthy politicians are 2020–1809 = 211 years old, 1809 being Abraham’s birth year. But what about politicians that have all the same accomplishments but are 211.001 and 210.999 years old?
Probably the same! But be extremely careful when using this technique since you are effectively falsifying your data. This technique can easily backfire, and you should only use it when you are very confident that it is the right choice. But this brings us to a slightly more elaborate and automated technique.
SMOTE stands for Synthetic Minority Over-sampling Technique. It constructs new samples by carefully observing the attributes of all samples. It then proceeds to alter the values just so that they stay within the range observed in our minority-class. Sounds complicated, let’s look at a simple example, and it should become as clear as your mouse clicks on the clap. Let’s assume we add JFK to the dataset and look again at the age of trustworthy politicians.
As we can see, the area indicated as SMOTE SAFE is the area in which we can freely generate new synthetic samples! You can find a more in-depth explanation here. It is fascinating, but I think it is most important to know that such methods exist.
We can find some incredible examples of this technique in the autonomous driving scene. While all the major competitors in this field have collected several petabytes of data, conditions such as kids running in front of cars should be tested before actually encountering them. These cases may also happen so rarely that you will never find enough training data.
Generating Synthetic data to test such and various other edge cases under many different lighting and weather conditions is a must. One often uses GAN’s or similar networks to convert already existing samples into various other forms, e.g., in winter or when raining. There are multiple ways of how this can be done. A short video that illustrates this can be seen here (I am not affiliated with them nor know how good they are, but they have an excellent short video ;)
But we do have one more thing! What if I told you that instead of dealing with the problem at the data level, we could deal with it at the heart of our algorithm, the loss function.
Over the years, many approaches have been proposed, and there truly is no one size fits all. It quite often depends heavily on how important it is to identify the minority-class, and what kind of costs occur when you misclassify a sample for the majority class.
Weighted Loss Function, Define It Yourself
One way of mixing such a loss is simply by multiplying the terms in which a minority-class occurs with a constant>1. What this effectively does is encourage the model to pay more attention to the instances of the minority class. In Keras this is built in an can be applied without additional hassle
class_weights = {0: 1.,                1: 50.}model.fit(X_train, Y_train,class_weights=class_weights)
In this particular case, it tells Keras that it should treat one occurrence of class 1 as 50 instances of class 0. In some easy python pseudo-code, we could think of it as something like this
def loss_weighted(x,y_true):y_pred=model.forward(x)    if y_true==1:        return 50*loss(y_pred,y_true)    elif y_true==0:        return 1*loss(y_pred,y_true)
As the amount of classes becomes bigger, it is tedious to defined things explicitly, as we did above. For this, a variety of solutions have been proposed, and one of the most notable works is “Class-Balanced Loss Based on Effective Number of Samples.”
What they effectively proposed is reweighting the loss function. The authors do this in such a way that the effective amount of samples per class are taken into account. To control on what side of the balancing the loss ends up, one can additionally tune it with a hyperparameter.
While this is a very simplistic view of their process, I feel that it grasped the most important aspects to keep in your memory. If you ever encounter a case where you have a lot of classes and significant imbalances between the classes, make sure to give this approach a go.
Congratulations, you know now what you can do when you encounter imbalanced data. Several approaches exist, and choosing the right one for your problem may not always be an easy task. It will heavily depend on your business needs to decide how much time and resources you want to put into balancing things more fairly. Being aware of potential solutions is the first step to solving them!
If you enjoyed this article, I would be excited to connect on Twitter or LinkedIn.
Make sure to check out my YouTube channel, where I will be publishing new videos every week.