K-Medoids Clustering on Iris Data Set
By: Tri Nguyen
Tags: Data Science, Towards Data Science, Clustering, Classification, Machine Learning

Pretty much in any machine learning course, K-Means Clustering would be one of the first algorithms to be introduced for unsupervised learning. Thanks to that, it has become much more popular than its cousin, K-Medoids Clustering. If you Google “k-means”, 1.49 billion results will pop up. Do that for “k-medoids”, only 231 thousand results return.

That was my struggle when I was asked to implement the k-medoids clustering algorithm during one of my final exams. It took a while but I managed it. Therefore, in order to help the concept of k-medoid see more action, I decided to put the algorithm that I built and implemented here, this time, based on the “notorious” Iris dataset.

Introduction

Both the k-means and k-medoids algorithms are partitional, which involves breaking the dataset into groups. K-means aims to minimize the total squared error from a central position in each cluster. These central positions are called centroids. On the other hand, k-medoids attempts to minimize the sum of dissimilarities between objects labeled to be in a cluster and one of the objects designated as the representative of that cluster. These representatives are called medoids.

In contrast to the k-means algorithm that the centroids are central, average positions that might not be data points in the set, k-medoids chooses medoids from the data points in the set.

Data Preparation

We will use a copy of Iris Data Set from the sklearn library . For the sake of algorithm demonstration, I will skip splitting the data set into training and testing sets and will use one single data set for training and fitting the model.

Below is the measurement data of the sepal length, width; petal length, width of 3 different types of iris: [‘setosa’, ‘versicolor’, ‘virginica’] labeled as [0, 1, 2]:

I have a habit of scaling the data for these types of problem when data is on different scales across the columns. Since there is no missing data, let’s proceed with the scaling. I utilized the MinMaxScaler class from the same sklearn library. Below is the scaled data:

There are 4 features which might not be ideal for visualization and fitting of the clustering model since some of them might be highly correlated. Therefore, we will employ the Principal Component Analysis (PCA) to transform 4-dimensional data into 3-dimensional data while keeping the significance of those predictors with the PCA class from sklearn. The dimensionally-reduced data is as below:

We can finally visualize these on a 3-D plane:

Implementation

We first start looking at the dissimilarity function. There are many choices but here, we will use the common Minkowski Distance Function of a given order. The algorithm that we will be implementing is called the Partitioning Around Medoids (PAM) algorithm (Kaufman and Rousseeuw 1990). It can be summarized as below:

Initialization: randomly select 𝑘 of the 𝑚 data points as the medoids Assignment: associate each data point to the closest medoid based on the Minkowski distance Update: for each medoid 𝑗 and each data point 𝑖 associated with 𝑗, swap 𝑗 and 𝑖 and compute the total cost of the configuration (which is, the average dissimilarity of 𝑖 to all the data points associated to 𝑗). Select the medoid 𝑗 with the lowest cost of the configuration. Iterate between steps 2 and 3 until there is no change in the assignments.

In the code that follows, it will be convenient to use our usual “data matrix” convention, that is, in the data matrix 𝑋, each row, 𝑥̂ , is one of 𝑚 observations and each column, 𝑥, is one of 𝑓 features.

1. Medoid Initialization

To start the algorithm, we need an initial guess. Let’s randomly choose 𝑘 observations from the data. In this case, 𝑘 = 3, representing 3 different types of iris.

Next, we will create a function, init_medoids(X, k), so that it randomly selects 𝑘 of the given observations to serve as medoids. It should return a NumPy array of size 𝑘×𝑑, where 𝑑 is the number of columns of X.

After initializing 3 random medoids from the data points, we have:

2. Computing the distances

Implementing a function that computes a distance matrix, 𝑆 = (s_𝑖𝑗) such that s_𝑖𝑗 = 𝑑^ 𝑝_𝑖𝑗 is the Minkowski distance of order 𝑝 from point 𝑥_𝑖 to medoid 𝜇_𝑖. It should return a NumPy matrix 𝑆 with shape (𝑚,𝑘).

The 𝑝^𝑡ℎ order Minkowski Distance between two points, 𝑥 and 𝜇 is given by:

Thus, for the elements of matrix 𝑆,

Minkowski distance is typically used with 𝑝 being 1 or 2, which correspond to the Manhattan distance and the Euclidean distance, respectively. We will use the Euclidean distance in our calculations.

Based on that, the first 5 elements of the matrix S are:

3. Cluster Assignment

Now we build a function that acts on the distance matrix S to assign a “cluster label” of 0, 1, 2 to each point using the minimum distance to find the “most similar” medoid.

That is, for each point, indicated by row index 𝑖 , if 𝑠_𝑖𝑗 is the minimum distance for point 𝑖, then the index 𝑗 is 𝑖 ‘s cluster label. In other words, the function should return a one-dimensional array, 𝑦, of length 𝑚 such that:

We then get the labels for the data points of the first iteration:

4. Swap Test

In this step, for each medoid 𝑗 and each data point 𝑖 associated to 𝑗 swap 𝑗 and 𝑖 and compute the total cost of the configuration (that is the average dissimilarity of 𝑖 to all the data points, ∑𝑠_𝑖𝑗) as Step 3. Select the medoid 𝑗 with the lowest cost of the configuration.

That is, for each cluster, search if any of the points in the cluster decreases the average dissimilarity coefficient. Select the point that decreases this coefficient the most as the new medoid for this cluster.

Lastly, we also need a function that checks whether the medoids have “moved,” given two instances of the medoid values. This is the function to check whether the medoids no longer move and the iteration should be stopped.

4. Putting all together

When we combined all the functions above, we essentially combine the steps of a K-Medoids Algorithm.

By running the function over the data points that we have, the final medoids are found to be:

And the cluster labels for each data points are:

By considering “inexact” matches, the K-Medoids Algorithm above labels correctly 94.7% of the data points. This is just for demonstration purposes of how the algorithm works and this number does not mean much as we did not split the data into training and testing sets.