Gaussian Mixture Models for Clustering
By: Vivienne Difrancesco
Tags: Gaussian Mixture Model, Data Science, Clustering

Gaussian Mixture Models for Clustering

Photo by Nareeta Martin on Unsplash

Introduction

Recently I was using K-Means in a project and decided to see what other options were out there for clustering algorithms. I always find it enjoyable to sink my teeth into expanding my data science skillset. I decided to write this article to share the experience of what I discovered on my quest to broaden my clustering knowledge to include using Gaussian Mixture Models.

An Overview of Gaussian Mixture Models

When hearing of this technique you may have thought about the Gaussian distribution (also called normal distribution). That’s exactly what this clustering technique is based on. It assumes that the data points come from multi-dimensional Gaussian distributions that could have varying parameters of covariance, mean, and density.

Gaussian Mixture models work based on an algorithm called Expectation-Maximization, or EM. When given the number of clusters for a Gaussian Mixture model, the EM algorithm tries to figure out the parameters of these Gaussian distributions in two basic steps.

The E-step makes a guess of the parameters based on available data. Data points are assigned to a Gaussian cluster and probabilities are calculated that they belong to that cluster.

Data points are assigned to a Gaussian cluster and probabilities are calculated that they belong to that cluster. The M-step updates the cluster parameters based on the calculations from the E-step. The mean, covariance, and density are calculated for clusters based on the data points in the E step.

The mean, covariance, and density are calculated for clusters based on the data points in the E step. The process is repeated with the calculated values continuing to be updated until convergence is reached.

If you are familiar with K-Means, this process at a high level is really the same. The similar flow being to make a guess, calculate values, and readjust until convergence.

Fitting a Gaussian Mixture Clustering Model

With a basic understanding of how Gaussian Mixture works, the fun part is to start coding and experimenting. You can implement a clustering model in just a few lines of code using Python and Scikit-Learn. I encourage you to look at the Scikit-Learn documentation page for the Gaussian Mixture class.

from sklearn.mixture import GaussianMixture gm = GaussianMixture(n_components=n, random_state=123, n_init=10)

preds = gm.fit_predict(X)

The n_components parameter is where you specify the number of clusters. The n_init parameter allows you to control how many times the algorithm is initialized. The initial placement of where clusters are located at the start can set the algorithm up for picking bad cluster parameters. By initializing multiple times, you decrease the chance of converging on bad clusters simply because of bad luck on initial placement. Finally, the fitting and predicting can be done in one step from calling the fit_predict() method on your data represented as X in the example.

How Many Clusters?

But how many clusters are in my data? You may have noticed that in order to apply the model to your data, you have to have a value chosen for the number of clusters. How do you know what cluster number best represents your data? You have a couple of options that I will go over.

Trial and Error

Often, the best way to find an appropriate cluster number is to try different cluster numbers and see which fits your data appropriately. The two most popular evaluation metrics for picking cluster numbers for fitting Gaussian Mixture models are BIC and AIC. BIC stands for Bayesian information criterion and AIC stands for Akaike information criterion. The intuition in the calculation of both of these metrics is that they favor the simplest model that maximize the likelihood function of the model.

Calculating the AIC and BIC is easy because they are built in as a method on the Scikit-Learn Gaussian Mixture class. By setting up a loop to try different cluster numbers and calculate the AIC and BIC, you can then plot the metrics together to pick your cluster number.

BIC and AIC plot (Image by the author)

You can see that the AIC and BIC mirror each other pretty closely. BIC and AIC are meant to be minimized so pick the low spot in the chart. The two measures will usually pick the same number for clusters, but if they differ then know that BIC more heavily favors simple models than AIC, but AIC models tend the fit the data a little better. In this example, I would pick 5 as the most appropriate cluster number for the data as the chart really levels off after that.

Bayesian Gaussian Mixture Models

Another method for picking the cluster number that I came across is by using the Bayesian Gaussian Mixture Models class in Scikit-Learn. When you fit this model it has an attribute to return the weights of clusters with erroneous clusters being weighted at or near zero and basically removing those clusters automatically. Once again you can fit this model in only a couple lines of code.

from sklearn.mixture import BayesianGaussianMixture

import numpy as np bgm = BayesianGaussianMixture(n_components=10, n_init=10)

bgm.fit(X)

np.round(bgm.weights_, 2)

The model is created and fit the same as before but now we can return an array of cluster weights. My own experiment returned this array:

array([0.09, 0.25, 0.16, 0.07, 0.06, 0.24, 0.13, 0., 0., 0. ])

This allows me to see that the model has basically picked the appropriate number of clusters automatically, as the ones that are not needed are weighted as 0.

Pros and Cons

One positive about these models is that they can handle a greater variety of shapes, primarily clusters that form ellipsis shapes. Something like K-Means is really only good at clusters that are roughly spherical. There is a covariance_type parameter that allows for adjusting for different shapes of the clusters. However, something like crescent shaped clusters will still be difficult for a Gaussian Mixture model to accurately identify.

Another positive is that these models allow for soft classification. K-Means is a hard classification model where each data point is assigned to a single cluster. The Gaussian Mixture method, however, calculates probabilities of data points belonging to clusters. These probabilities can be obtained using the predict_proba() method after fitting the model.

A caveat I found when playing with Gaussian Mixture models is that they take a lot longer to run than a K-Means model because they take a lot longer to converge. Be mindful of the amount of data you are using and how many features are involved. When experimenting with the Bayesian Gaussian Mixture class specifically, it often gave errors about how convergence couldn’t be reached. When this happens you can increase the max_iter parameter, but that also means it will take longer to run. Keep in mind that it also may mean that your data is not well suited to a Gaussian Mixture clustering style.

Conclusion

There is so much more to explore with this style of clustering. I tried to have this be a guide that explained concepts in plain English to get you started, but I encourage you to look up formulas and dig into the math behind what is happening. The Scikit-Learn clustering user guide is a great place to start.