Imbalanced Data — Oversampling Using Gaussian Mixture Models | by Bassel Karami | Towards Data Science

TL;DR — Drawing samples from Gaussian Mixture Models (GMM), or other generative models, is another creative oversampling technique that can potentially outperform SMOTE variants.
Introduction
Dataset preparation
Intro to GMM
Using GMM as an oversampling technique
Evaluation of performance metrics
Conclustion
In a previous article, I discuss how one can come up with many creative oversampling techniques that can outperform SMOTE variants. We saw how oversampling using “crossovers” outperformed SMOTE variants.
In this article, we will show how Gaussian Mixture Models (GMM), or generative model, can be used to oversample minority class instances in an imbalanced dataset.
We first generate an imablanced classification dataset using scikit-learn’s make_classification function with 5000 data points across two classes (binary classification). There is a 95% chance the target is 0 and a 5% chance the target is 1. We prepare a similar dataset to the previous article.
from sklearn.datasets import make_classificationimport seaborn as snsX, y = make_classification(    n_samples=5000, n_classes=2,     weights=[0.95, 0.05], flip_y=0)sns.countplot(y)plt.show()
By default 20 features are created, below is what a sample entry in our X array looks like.
Finally, we split the data into training and testing datasets.
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y)
Gaussian mixture models (GMM) assume that there are a number of normally distributed subpopulations that the data comes from.
Example: Suppose we have data on the height of 10,000 individuals. Below is a sample distribution. It doesn’t look like a bell-shaped normal distribution.
However, it actually comes from 4 different groups of people with different average heights that are normally distributed. One group has an average height of 150 cm, the others have average heights of 160, 170, and 180 cm.
Below is the code used to generate the chart. You can see the 4 different groups out of which we generate 4000, 1000, 1000, and 4000 sample data respectively, totaling 10,000 sample individuals.
samples_150 = ss.norm(150, 10).rvs(4000)samples_160 = ss.norm(160, 10).rvs(1000)samples_170 = ss.norm(170, 10).rvs(1000)samples_180 = ss.norm(180, 10).rvs(4000)samples_total = np.hstack([samples_150, samples_160,                          samples_170, samples_180])plt.figure(figsize=(10, 5))ax = sns.distplot(samples_total)ax.set_xlabel("Height (in cm)")plt.show()
The height in each group is normally distributed.
plt.figure(figsize=(10, 5))ax = sns.distplot(samples_150)ax.set_xlabel("Height (in cm)")plt.show()
The idea behind GMM is that a dataset may have observations coming from different subpopulations that have their own characteristics. GMM helps in extracting the subpopulations from the data such that each subpopulation forms a cluster that we can use.
If we go back to the classification dataset we generated in the section “Dataset Preparation”, we can try to extract subpopulations/clusters from the data. Let’s extract 5 clusters for example.
gmm = GaussianMixture(5)gmm.fit(X_train)
That was it!
Under the hood, our GMM model has now created 5 different clusters with different normal distributions reflecting the feature values each cluster can take.
Below is an example showing the cluster mean values for each of the 20 features.
pd.DataFrame(gmm.means_.T)
More importantly, the GMM model can help us with 2 functions:
1- It can look at the feature values for a particular sample and assign the sample to a cluster.
gmm.predict(X_test)
2- It can use the fitted normal distributions to generate new samples which we can use for oversampling.
gmm.sample(5)
Finally, in this example, we are clustering our data into 5 different buckets, but our problem is a binary classification problem where our target variable can be either 0 or 1.
One idea is to check for the relationship between every cluster and the target (y) variable.
cluster_mean = pd.DataFrame(data={    "Cluster": gmm.predict(X_train),     "Mean Target Variable (y)": y_train}).groupby("Cluster").mean().reset_index(drop=False)plt.figure(figsize=(10, 5))sns.barplot(data=cluster_mean, x="Cluster",      y="Mean Target Variable (y)")plt.show()
We can see that Cluster 4 has the highest mean value for the target value at just over 40%.
Remember this is an imbalanced dataset where only 5% of the samples have a target variable (y) value = 1, so 40% is a large number.
The last step would be to generate random samples from the GMM model and only keep the ones which belong to Cluster 4. We can label them with a positive target variable (y=1).
samples, clusters = gmm.sample(100)samples_to_keep = samples[clusters==4]
We can finally add them to our training data!
We can similarly draw sample from the top 2–3 clusters that are most strongly associated with y=1. Or we can draw sample from any cluster where the mean value of y is above a predefined threshold.
When testing the performance of this oversampling technique we will use the below function which summarizes all the steps we discussed.
def oversample_gmm(X, y, rows_1, no_clusters=2,                    no_sampling_clusters=1, rs=1):    # Instantiate GMM model    gmm = GaussianMixture(        no_clusters,        covariance_type='tied',         max_iter=10000,        random_state=rs    )    gmm.fit(X)        # Finding cluster with y=1 most likely    cluster_mean = pd.DataFrame(data={        "Cluster": gmm.predict(X_train),         "Mean Target Variable (y)": y_train        }).groupby("Cluster").mean().sort_values(        by='Mean Target Variable (y)', ascending=False)top_clusters = cluster_mean.sort_values(        by='Mean Target Variable (y)', ascending=False).index[        :no_sampling_clusters]        # Number of rows we will generate before    # filtering by required cluster.    # Multiply by 5 to ensure we have sufficient samples    rows_initial = rows_1 * no_clusters * 5        # Generate samples    samples, clusters = gmm.sample(rows_initial)# Keep samples coming from clusters where y=1 is likely    top_clusters_filter = [np.any([        cluster == x for x in top_clusters]        ) for cluster in clusters]    samples_to_keep = samples[top_clusters_filter]        # Keep only required number of additional samples    rows_required = rows_1 - np.sum(y)    np.random.shuffle(samples_to_keep)    samples_to_keep = samples_to_keep[:rows_required]        # Add samples to training dataset    X_gmm = np.vstack([X, samples_to_keep])    y_gmm = np.hstack([y, np.ones(samples_to_keep.shape[0])])        return X_gmm, y_gmm
Similar to our previous article, we iterate through 30 random states and compare the performance of a random forest classifier on the original dataset as well as the oversampling methods, ensuring we have 2000 training samples with a positive target value (target = 1):
Random oversampling
SMOTE — 1 neighbor
SMOTE — 3 neighbors
SMOTE — 5 neighbors
SMOTE — 10 neighbors
GMM — 2
GMM — 3
GMM — 5
GMM — 10
GMM — 15
We also look into 7 classification metrics:
ROC AUC — area under the ROC curve
PR AUC — area under the precision-recall curve
Balanced accuracy — this is also equivalent to macro-averaged recall across both labels
Max F1 — Maximum F1 score for different probability cutoffs (i.e. if we predict 1 when predicted probability > 0.2 or 0.3 etc. instead of the default 0.5)
Recall
Precision
F1 score
Below are the results...
There are mixed results of course.
GMM with 10 clusters is the best option when looking at the metrics that are arguably most relevant for imbalanced datasets: Precision-Recall curve, F1 score, and Max F1 (an alternative to F1 where we consider all probability thresholds).
We’ve demonstrated how another creative, yet simple, oversampling technique for imbalanced datasets can outperform default techniques such as SMOTE and random oversampling on several metrics.
The same GMM-based approach can also be used for generating synthetic data samples to regularize your predition model. This is particularly helpful in use cases where a lot of model/data drift is expected and cross-validation results are misleading.
That is a topic to discuss in a subsequent post...