Oversampling and Undersampling. A technique for Imbalanced... | by Kurtis Pykes | Towards Data Science

The Imbalanced classification problem is what we face when there is a severe skew in the class distribution of our training data. Okay, the skew may not be extremely severe (it can vary), but the reason we identify imbalanced classification as a problem is because it can influence the performance on our Machine Learning algorithms.
One way the imbalance may affect our Machine Learning algorithm is when our algorithm completely ignores the minority class. The reason this is an issue is because the minority class is often the class that we are most interested in. For instance, when building a classifier to classify fraudulent and non-fraudulent transactions from various observations, the data is likely to have more non-fraudulent transactions than that of fraud — I mean think about it, it would be very worrying if we had an equal amount of fraudulent transactions as non-fraud.
An approach to combat this challenge is Random Sampling. There are two main ways to perform random resampling, both of which have there pros and cons:
Oversampling — Duplicating samples from the minority class
Undersampling — Deleting samples from the majority class.
In other words, Both oversampling and undersampling involve introducing a bias to select more samples from one class than from another, to compensate for an imbalance that is either already present in the data, or likely to develop if a purely random sample were taken (Source: Wikipedia).
We define Random Sampling as a naive technique because when performed it assumes nothing of the data. It involves creating a new transformed version of our data in which a there is a new class distribution to reduce the influence of the data on our Machine Learning algorithm.
Note: We refer to Random Resampling as naive because when performed it makes no assumptions of the data.
In this article we will be leveraging the imbalanced-learn framework which was initiated in 2014 with the main focus being on SMOTE (another technique for imbalanced data) implementation. Over the years, additional oversampling and undersampling methods have been implemented as well as making the framework compatible with the popular machine learning framework scikit-learn. Visit Imbalanced-Learn for guides on installation and the full documentation.
from sklearn.datasets import make_classificationfrom imblearn.over_sampling import RandomOverSamplerfrom imblearn.under_sampling import RandomUnderSamplerfrom collections import Counter# defining the datasetX, y = make_classification(n_samples= 10000, weights=[.99])# class distributionprint(Counter(y))Counter({0: 9844, 1: 156})
For the full code you may visit my Github.
github.com
Random Oversampling includes selecting random examples from the minority class with replacement and supplementing the training data with multiple copies of this instance, hence it is possible that a single instance may be selected multiple times.
“the random oversampling may increase the likelihood of overfitting occurring, since it makes exact copies of the minority class examples. In this way, a symbolic classifier, for instance, might construct rules that are apparently accurate, but actually cove one replicated example.” — Page 83, Learning from Imbalanced Data Sets, 2018.
For Machine Learning algorithms affected by skewed distribution, such as artificial neural networks and SVMs, this is a highly effective technique. However, tuning the target class distribution is advised in many scenarios as seeking a balanced distribution for a severely imbalanced dataset can lead to the algorithm overfitting the minority class, in turn resulting in an increase of our generalization error.
Another thing we ought to be aware of is the increased computational cost. Increasing the number of examples in the minority class (especially for a severely skewed data set) may result in an increased computational when we train our model and considering the model is seeing the same examples multiple times, this isn’t a good thing.
Nonetheless, Oversampling is a pretty decent solution and should be tested. Here is how we can implement it in Python...
# instantiating the random over sampler ros = RandomOverSampler()# resampling X, yX_ros, y_ros = ros.fit_resample(X, y)# new class distribution print(Counter(y_ros))Counter({0: 9844, 1: 9844})
Random Undersampling is the opposite to Random Oversampling. This method seeks to randomly select and remove samples from the majority class, consequently reducing the number of examples in the majority class in the transformed data.
“In random under-sampling (potentially), vast quantities of data are discarded. [...] This can be highly problematic, as the loss of such data can make the decision boundary between the minority and majority instances harder to learn, resulting in a loss in classification performance.” — Page 45, Imbalanced Learning: Foundations, Algorithms and Applications, 2013
The result of undersampling is a transformed data set with less examples in the majority class — this process may be repeated until the number of examples in each class is equal.
Using this approach is effective in situations where the minority class has a sufficient amount of examples despite the severe imbalance. On the other hand, it is always important to consider the prospects of valuable information being deleted as we randomly remove them from our data set since we have no way to detect or preserve the examples that are information rich in the majority class.
To better understand this method, here is a python implementation...
# instantiating the random undersamplerrus = RandomUnderSampler() # resampling X, yX_rus, y_rus = rus.fit_resample(X, y)# new class distributionprint(Counter(y_rus))Counter({0: 156, 1: 156})
Combining both random sampling methods can occasionally result in overall improved performance in comparison to the methods being performed in isolation.
The concept is that we can apply a modest amount of oversampling to the minority class, which improves the bias to the minority class examples, whilst we also perform a modest amount of undersampling on the majority class to reduce the bias on the majority class examples.
To implement this in Python, leveraging the imbalanced-learn framework, we may the sampling_strategy attribute in our oversampling and undersampling techniques.
# instantiating over and under samplerover = RandomOverSampler(sampling_strategy=0.5)under = RandomUnderSampler(sampling_strategy=0.8)# first performing oversampling to minority classX_over, y_over = over.fit_resample(X, y)print(f"Oversampled: {Counter(y_over)}")Oversampled: Counter({0: 9844, 1: 4922})# now to comine under sampling X_combined_sampling, y_combined_sampling = under.fit_resample(X_over, y_over)print(f"Combined Random Sampling: {Counter(y_combined_sampling)}")Combined Random Sampling: Counter({0: 6152, 1: 4922})
In this guide we discussed oversampling and undersampling for imbalanced classification. There are various occasions where we may be confronted with an imbalanced dataset and applying random sampling may provide us with a very good model to overcome this problem in training and still maintain a model that generalizes well to new examples.
Let’s continue the conversation on LinkedIn...