K-Means Clustering and Gaussian Mixture Models
By: Jenna Jones
Tags: Towards Data Science, Unsupervised Learning, Gaussian Mixture Model, Clustering, K Means

Clustering: Out of the Black Box

Photo by Mick Haupt on Unsplash

By: Nick Paine and Jenna Jones

Introduction

Recently, unsupervised learning has received a lot of attention due to its ability to find patterns in data without any training. Specifically, clustering has been used to solve many data problems, including customer segmentation, fraud detection, recommendation engines and most importantly, football player positions. In this paper, we discuss and compare two different methods for grouping data points together: K-Means and Gaussian Mixture Models (GMM). We then use each of these methods to cluster the same example set of data.

Terminology

As always with machine learning systems, some special language is needed:

Features : measurable data that can be used for analysis (if we have a data set of football players, we could include Height as a feature, Weight as a feature, college as a feature).

: measurable data that can be used for analysis (if we have a data set of football players, we could include Height as a feature, Weight as a feature, college as a feature). Number of dimensions : the number of features in a data set. In our example, there will be two dimensions: Height and Weight of each player. It is easy to visualize two dimensions (typically seen in a math class) and it is possible to visualize three dimensions, but after that, higher dimensions can not easily be visualized. Despite this, there can be an unlimited number of dimensions in a data set.

: the number of features in a data set. In our example, there will be two dimensions: Height and Weight of each player. It is easy to visualize two dimensions (typically seen in a math class) and it is possible to visualize three dimensions, but after that, higher dimensions can not easily be visualized. Despite this, there can be an unlimited number of dimensions in a data set. Centroid : the calculated center position of a cluster of data points. For K-Means, this is the arithmetic mean of data points in each cluster for each dimension.

: the calculated center position of a cluster of data points. For K-Means, this is the arithmetic mean of data points in each cluster for each dimension. Cluster : a set of data points that are grouped together by similar features.

: a set of data points that are grouped together by similar features. Distance measure : the method of calculating how far away any data point is from each cluster centroid.

: the method of calculating how far away any data point is from each cluster centroid. Sum of Squared Errors: once the data has been segregated into a given number of clusters, this measure takes the sum of the squared distance between each data point and its cluster center adding up all of the data points in the analysis. It is used to help assess how many clusters there are in the data.

Example

Data: We use a 2-dimensional set of data that shows the Heights and Weights of NFL football players classified as running backs, tight ends and defensive tackles in the AFC and NFC South divisions. There are 94 data points.

An extract of the data set which shows the data in two dimensions

Goal: to correctly assign each player into one of K clusters (positions) based on Height and Weight. Keep in mind that the algorithm has seen no labeled data nor had any training - the decision will be made solely based on the two dimensional data.

Below is a plot of the raw data, where each point represents a player. Can you visualize the optimal number of clusters?

scatter plot of raw data

This is quite a tough problem for the computer as there are overlapping data points and some outliers.

A visual examination appears to show that there are two distinct clusters: one located at a higher Weight range and one spanning the lower Weight range inclusive of various Heights. It is difficult to say whether there is only one or many clusters within this second group simply by looking.

We want to evaluate how successfully K-Means and GMM can group the data. Let’s have a look at the actual clusters before we get started.

The actual clusters

K-Means Clustering

K-Means is easily the most popular clustering algorithm due to its simplicity. Ultimately, it assumes that the closer data points are to each other, the more similar they are. Here is how it is done:

Choose the number of clusters K Randomly establish the initial position for each centroid Assign each data point to the nearest centroid using the distance measure Reset each centroid location using the mean of the data points assigned to each cluster Return to step 3 and repeat until no single data point changes cluster

How do you find the optimal number of clusters, K?

One of the most difficult steps in clustering is to determine the optimal number of clusters, K, to group the data, and there is no ‘right’ answer. The most common approach is known as ‘the elbow method’. Essentially, it runs K-Means clustering on the whole data set for various values of K and calculates the overall sum of squared errors (SSE) for each K.

It is important to plot the results. The plot resembles an arm, and as the name suggests, the value of K where the steepness changes the most (ie the elbow) is considered the optimal number of clusters. The objective is not to find the K that minimizes total squared distance, but rather the K that sees diminishing returns when K is increased.

In our example, the elbow method returns a value of 3 as the optimal number of clusters (and we know this to be correct based on the actual data).

What is the distance measure?

The K-Means algorithm uses the Euclidean Distance Measure. This means that the measure of distance around each cluster center is ‘circular’. What this means is that the importance of each dimension is equal, hence the term ‘circular’.

where J is the number of dimensions.

As in our example, applying this formula to two dimensions, we take:

So, if we compute the distance between the Height and Weight of Derrick Henry (DH) versus the Heights and Weights of the RB and TE centroids, we see that there is a much shorter distance to the TE centroid and therefore, he is classified as a TE.

How do you reset each centroid location?

Having assigned each data point to a cluster, we simply calculate the arithmetic mean of the data points for each dimension in each cluster (ie the mean of all player Heights and the mean of all player Weights for each position). This is then the new position of each centroid. The use of the mean is where the algorithm gets its name: K-Means.

The algorithm is implemented directly in the sklearn.cluster.KMeans package. However, we will go through a small example to show what happens behind the scenes.

To start, in step 2, each centroid is randomly initialized. In the second image, step 3, the data points are assigned to the centroid that is closest using the distance measure.

Next, each centroid location is reset based on the mean. In the second image, the data points are reassigned to the centroid that now has the least distance to the data point.

The process above is repeated

The process is finished since no data point changes clusters with the new centroid locations

We ran the algorithm on the data set in Python and the results were encouraging.

K-Means successfully clustered the data into three groups; however, it mis-assigned three RBs as TEs.

Results: The clusters correspond to the actual data clusters with only three players being misclassified. We are able to easily evaluate the clustering since we already know each players position. All DTs were clustered into one group successfully, but that is not surprising given that we could see that by eye. What is impressive is that the majority of the RBs and TEs were clustered correctly despite appearing to be one continuous group of data.

Feature Standardization

When features have different scales (such as housing prices- 400,000 and square feet- 1,500), it can be important to preprocess the data by standardizing it so that, in this example, the contribution of housing prices to the distance isn’t overly huge compared to that of the square feet and therefore dominating the whole distance measure. In order to more equally distribute the contributions between features, we can standardize the data so that the spread around the mean for each dimension of the whole data set is the same. In our example, we have chosen not to standardize the data as we believe that the Weight of the players is naturally more important than their Height. However, to illustrate the effect of standardization, we show the results of the analysis of our example data with and without feature standardization.

Cross tabulation of the assigned clusters from K-Means and the actual data groupings. Clusters 0 and 1 correspond perfectly with the DT and RB positions. Cluster 2 includes mainly TEs and incorrectly includes 3 RBs.

Cross tabulation of the assigned clusters as shown above with feature standardization

With standardized data, there are fewer RBs misclassified. However, more DTs are misclassified because the standardization squashes the weight range and brings the two ~300 lb players into the TE group.

K-Means with feature standardization

As we can see, the effects of feature standardization will depend on the data and the make-up of the structure and size of features.

Advantages of K-Means:

Simple to understand

Very quick (all that is being computed is the distance between each point and cluster center)

Easy to implement

Disadvantages of K-Means:

Have to choose K manually

Depends on initial centroid locations

Potential to misrepresent centroid positions due to outliers

Gaussian Mixture Models

The Gaussian Mixture Model is an Expectation-Maximization (EM) algorithm with data points that are assumed to have a Gaussian (Normal) distribution. It is commonly described as a more sophisticated version of K-Means. It requires two parameters, the mean and the covariance, to describe the position and shape of each cluster.

The model is based on the multivariate Gaussian distribution which is:

where for each cluster:

GMM requires a five step process:

Choose the number of clusters, K Randomly establish the initial parameters for each centroid (ie the mean, covariance and weighting) Calculate the likelihood that the data set has come from this set of clusters with the current parameters Maximize the likelihood by changing the parameters (cluster proportions, centers and spread around the centers) Return to step 3 and repeat until the change in the parameters is less than a pre-programmed amount or a fixed number of iterations have been completed

What is the Expectation-Maximization algorithm?

EM starts by calculating the likelihood that a given data point comes from the mixture of Gaussian distributions with the current set of parameters. It then maximizes the likelihood by changing the parameters.

What is the likelihood and how is it maximized?

The likelihood function is based on the probability that a data point comes from the weighted collection of Gaussian clusters:

where:

Of course, the weights will sum to 1.

The likelihood is then the probability of the whole data set of N data points coming from the set of clusters with these parameters.

In common with many likelihood maximization problems, it is easier to take logs and maximize the log-likelihood function (which gives the same maximum or minimum values) because the product in the likelihood function becomes a sum when logs are taken. By differentiating with respect to each parameter (being cluster weighting, cluster mean and cluster covariance), the formula for updating each parameter is fairly easily obtained. The exact process is beyond the scope of this paper, but the resulting formulae are given below.

Expectation: without assigning a data point to an individual cluster, we calculate the probability that the parameters of each cluster will generate each data point. This means re-calculating for each iteration a new value of P(x) as given above.

Maximization: we need to update the weight, mean and covariance for each cluster respectively, using the following formulae:

What is the distance measure?

GMM does not use a distance measure, but applies a probability distribution around the cluster centers to work out the likelihood that a data point belongs to a given cluster. From the name, this is a multivariate normal distribution with the number of variates equal to the number of dimensions in the data.

Since GMM is finding the optimum parameters for each cluster, we may ultimately wish to assign each data point to a cluster. This is done by selecting the centroid ‘nearest’ to each data point. To do this, the Sklearn package from Python uses a distance measure called the Mahalenobis distance rather than the Euclidean distance used in K-Means. This measure is defined as:

It is clear that this formula allows for ellipsoidal contours around centroids rather than circular ones and its form is the same as that used in the multivariate Gaussian distribution formula.

Running the algorithm on our data set gave slightly better results than K-Means.

GMM successfully clustered the data into three groups; however, it mis-assigned one RB and one TE.

Results: All DTs were again clustered into one group successfully while Derrick Henry was again misclassified as a Tight End and a Tight End was misclassified as a Running Back.

Cross tabulation of the assigned cluster from GMM and the actual data groupings. Cluster 0 corresponds perfectly with the DT position. Clusters 1 and 2 are mainly correct but incorrectly include 1 player

Advantages of GMM:

can analyze more complex and mixed data

can handle outliers more easily

Disadvantages of GMM:

more difficult to directly interpret results

does not directly assign data points to clusters

Comparison and Discussion of the Two Methods

1: Similarities

Selection of the Starting Points of centroids

This is very important as it can have a big effect on the outcome.

Various methods have been proposed to initialize the cluster centers. Common methods are:

Forgy (chooses K data points randomly and use those as the initial centroid locations)

random partition (randomly assigns a cluster to each data point and then calculates a centroid based on this clustering- this generally creates initial centroids close to the middle of the data set)

K-Means++ (chooses initial centroids that are as far apart from each other as possible, effectively around the edges of the data set).

For GMM, it is usual to set the initial spread (variance) for each cluster as the population variance and set the weights equally.

Both Sklearn packages give the option to run the analysis a number of times using different starting points and give a result that is the best fit of all the runs. The number of different tries at running the analysis can be set by the user.

Tendency towards local minima rather than global minimum

Because the initialization of the centroids is essentially a guess, they can start far away from the true cluster centers in the data. The two methods always converge (meaning they will always find a solution) but not necessarily the optimal one. On the way, they may fall into a local minimum get stuck there.

an example of a function with two local minima and a global minimum

When the global minimum is not reached, the result will not be optimal and a different group of clusters will be obtained.

The results show the algorithm reaching a global minimum on the left and a local minimum on the right. Both are solutions but the global minimum is a better clustering of the data

2: Differences

Deterministic vs. Probabilistic Approach

K-Means: uses a deterministic approach and assigns each data point to one unique cluster (ex: Derrick Henry = TE). This is referred to as a hard clustering method.

and assigns each data point to one unique cluster (ex: Derrick Henry = TE). This is referred to as a hard clustering method. GMM: uses a probabilistic approach and gives the probability of each data point belonging to any of the clusters (ex: Derrick Henry = [0.63 TE, 0.33 RB, .04 DT]. This is referred to as a soft clustering method.

Parameters

K-Means: only uses two parameters : the number of clusters K and the centroid locations

: the number of clusters K and the centroid locations GMM: uses three parameters: the number of clusters K, mean, and cluster covariances

Updating the centroids

K-Means: uses only the data points assigned to each specific cluster to update the centroid mean

GMM: uses all data points within the data set to update the centroid weights, means and covariances

Feature Standardization

K-Means: the ‘circular’ distance measure around a centroid can make feature standardization necessary if one or more of the dimensions will dominate the calculation

GMM: automatically takes the issue into account by its calculation and use of the covariance matrix

Python

Python implements both clustering techniques through its Sklearn package. The packages are fairly easy to use and retain a great deal of information about the data. They also allow for a number of user choices, defined by the hyper-parameters that you add to the statements.

Hyper-parameters for the Python code

Number of clusters (n_clusters): the number of clusters to group data into as well as the number of centroids

Maximum iterations (max_iter): there are two ways for the algorithm to finish: one is when the data points stop changing cluster (as in the image above) and the other is to set a fixed number of cycles for the algorithm (this will end the algorithm sooner than a convergence)

Number initial (n_init): number of times the whole algorithm is run, with different starting locations for the centroids. Since the algorithm is sensitive to starting points, the package allows you to select a number of different starting points with a view of seeking the optimal solution.

Random State (random_state): sets the same position for every run when initializing centroids

Once you have a Data Frame (called ‘data’ in the code below) that only contains the feature data, you can easily run K-Means and GMM Clustering with very little code.

Conclusion

We have used K-Means and GMM to cluster a data set using standard Python packages. We hope that this paper will help you to analyze your data in a better way by understanding how these algorithms work.

You may need to try a number of different approaches to get the optimum result: use different hyper-parameters, adjust the number of clusters, apply different distance measures and possibly standardize your data.

Clustering will not be simple or obvious in most real-life situations and so a deeper understanding of these tools, the way they can be used and the options available to the data scientist will lead to a superior result.

All non-referenced images were created by the authors

References

[1] Mahalanobis, On the Generalized Distance in Statistics, (1936), Proceedings of the National Institute of Sciences of India 2: 49–55

[2] Hilpisch, Y., Python for Finance, (2019), O’Reilly Media Inc.

[3] Pedregosa et al., Scikit-learn: Machine Learning in Python, (2011), JMLR 12, pp. 2825–2830, 2011