Clustering Using Convex Hulls. How to use convex hulls in data... | by Vijini Mallawaarachchi | Towards Data Science

I recently came across the article titled High-dimensional data clustering by using local affine/convex hulls by Hakan Cevikalp in Pattern Recognition Letters. It proposes a novel algorithm to cluster high-dimensional data using local affine/convex hulls. I was inspired by their method of using convex hulls for clustering. I wanted to give a try at implementing my own simple clustering approach using convex hulls. So, in this article, I will walk you through my implementation of my clustering approach using convex hulls. Before we get into coding, let’s see what a convex hull is.
According to Wikipedia, a convex hull is defined as follows.
In geometry, the convex hull or convex envelope or convex closure of a shape is the smallest convex set that contains it.
Let us consider an example of a simple analogy. Assume that there are a few nails hammered half-way into a plank of wood as shown in Figure 1. You take a rubber band, stretch it to enclose the nails and let it go. It will fit around the outermost nails (shown in blue) and take a shape that minimizes its length. The area enclosed by the rubber band is called the convex hull of the set of nails.
This convex hull (shown in Figure 1) in 2-dimensional space will be a convex polygon where all its interior angles are less than 180°. If it is in a 3-dimensional or higher-dimensional space, the convex hull will be a polyhedron.
There are several algorithms that can determine the convex hull of a given set of points. Some famous algorithms are the gift wrapping algorithm and the Graham scan algorithm.
Since a convex hull encloses a set of points, it can act as a cluster boundary, allowing us to determine points within a cluster. Hence, we can make use of convex hulls and perform clustering. Let’s get into the code.
I will be using Python for this example. Before getting started, we need the following Python libraries.
sklearnnumpymatplotlibmpl_toolkitsitertoolsscipyquadprog
To create our sample dataset, I will be using sci-kit learn library’s make blobs function. I will make 3 clusters.
import numpy as npfrom sklearn.datasets import make_blobscenters = [[0, 1, 0], [1.5, 1.5, 1], [1, 1, 1]]stds = [0.13, 0.12, 0.12]X, labels_true = make_blobs(n_samples=1000, centers=centers, cluster_std=stds, random_state=0)point_indices = np.arange(1000)
Since this is a dataset of points with 3 dimensions, I will be drawing a 3D plot to show our ground truth clusters. Figure 2 denotes the scatter plot of the dataset with coloured clusters.
import matplotlib.pyplot as pltfrom mpl_toolkits.mplot3d import Axes3Dx = X[:,0]y = X[:,1] z = X[:,2]# Creating figure fig = plt.figure(figsize = (15, 10)) ax = plt.axes(projection ="3d")     # Add gridlines  ax.grid(b = True, color ='grey',          linestyle ='-.', linewidth = 0.3,          alpha = 0.2)    mycolours = ["red", "green", "blue"]# Creating color map col = [mycolours[i] for i in labels_true]# Creating plot sctt = ax.scatter3D(x, y, z, c = col, marker ='o')plt.title("3D scatter plot of the data\n") ax.set_xlabel('X-axis', fontweight ='bold')  ax.set_ylabel('Y-axis', fontweight ='bold')  ax.set_zlabel('Z-axis', fontweight ='bold')  # show plot plt.draw()
First, we need to break our dataset into 2 parts. One part will be used as seeds to obtain an initial clustering using K-means. The points in the other part will be assigned to clusters based on the initial clustering.
from sklearn.model_selection import train_test_splitX_seeds, X_rest, y_seeds, y_rest, id_seeds, id_rest = train_test_split(X, labels_true, point_indices, test_size=0.33, random_state=42)
Now we perform K-means clustering on the seed points.
from sklearn.cluster import KMeanskmeans = KMeans(n_clusters=3, random_state=9).fit(X_seeds)initial_result = kmeans.labels_
Since the resulting labels may not be the same as the ground truth labels, we have to map the two sets of labels. For this, we can use the following function.
from itertools import permutations# Source: https://stackoverflow.com/questions/11683785/how-can-i-match-up-cluster-labels-to-my-ground-truth-labels-in-matlabdef remap_labels(pred_labels, true_labels):    pred_labels, true_labels = np.array(pred_labels), np.array(true_labels)    assert pred_labels.ndim == 1 == true_labels.ndim    assert len(pred_labels) == len(true_labels)    cluster_names = np.unique(pred_labels)    accuracy = 0    perms = np.array(list(permutations(np.unique(true_labels))))    remapped_labels = true_labels    for perm in perms:        flipped_labels = np.zeros(len(true_labels))        for label_index, label in enumerate(cluster_names):            flipped_labels[pred_labels == label] = perm[label_index]        testAcc = np.sum(flipped_labels == true_labels) / len(true_labels)        if testAcc > accuracy:            accuracy = testAcc            remapped_labels = flipped_labels    return accuracy, remapped_labels
We can get the accuracy and the mapped initial labels from the above function.
intial_accuracy, remapped_initial_result = remap_labels(initial_result, y_seeds)
Figure 3 denotes the initial clustering of the seed points.
Once we have obtained an initial clustering, we can get the convex hulls for each cluster. First, we have to get the indices of each data point in the clusters.
# Get the idices of the data points belonging to each clusterindices = {}for i in range(len(id_seeds)):    if int(remapped_initial_result[i]) not in indices:        indices[int(remapped_initial_result[i])] = [i]    else:        indices[int(remapped_initial_result[i])].append(i)
Now we can obtain the convex hulls from each cluster.
from scipy.spatial import ConvexHull# Get convex hulls for each clusterhulls = {}for i in indices:    hull = ConvexHull(X_seeds[indices[i]])    hulls[i] = hull
Figure 4 denotes the convex hulls representing each of the 3 clusters.
Now that we have the convex hulls of the initial clusters, we can assign the remaining points to the cluster of the closest convex hull. First, we have to get the projection of the data point on to a convex hull. To do so, we can use the following function.
from quadprog import solve_qp# Source: https://stackoverflow.com/questions/42248202/find-the-projection-of-a-point-on-the-convex-hull-with-scipydef proj2hull(z, equations):    G = np.eye(len(z), dtype=float)    a = np.array(z, dtype=float)    C = np.array(-equations[:, :-1], dtype=float)    b = np.array(equations[:, -1], dtype=float)    x, f, xu, itr, lag, act = solve_qp(G, a, C.T, b, meq=0, factorized=True)    return x
The problem of finding the projection of a point on a convex hull can be solved using quadratic programming. The above function makes use of the quadprog module. You can install the quadprog module using conda or pip.
conda install -c omnia quadprogORpip install quadprog
I won’t go into details about how to solve this problem using quadratic programming. If you are interested, you can read more from here and here.
Once you have obtained the projection on the convex hull, you can calculate the distance from the point to the convex hull as shown in Figure 5. Based on this distance, now let’s assign the remaining data points to the cluster of the closest convex hull.
I will consider the Euclidean distance from the data point to its projection on the convex hull. Then the data point will be assigned to the cluster with the convex hull having the shortest distance from that data point. If a point lies within the convex hull, then the distance will be 0.
prediction = []for z1 in X_rest:    min_cluster_distance = 100000    min_distance_point = ""    min_cluster_distance_hull = ""        for i in indices:        p = proj2hull(z1, hulls[i].equations)        dist = np.linalg.norm(z1-p)        if dist < min_cluster_distance:            min_cluster_distance = dist            min_distance_point = p            min_cluster_distance_hull = i    prediction.append(min_cluster_distance_hull)prediction = np.array(prediction)
Figure 6 denotes the final clustering result.
Let’s evaluate our result to see how accurate it is.
from sklearn.metrics import accuracy_scoreY_pred = np.concatenate((remapped_initial_result, prediction))Y_real = np.concatenate((y_seeds, y_rest))print(accuracy_score(Y_real, Y_pred))
I got an accuracy of 1.0 (100%)! Awesome and exciting right? 😊
If you want to know more about evaluating clustering results, you can check out my previous article Evaluating Clustering Results.
towardsdatascience.com
I have used a very simple dataset. You can try this method with more complex datasets and see what happens.
I also tried to cluster a dataset with data points having 8 dimensions using my cluster hull method. You can find the jupyter notebook showing the code and results. The final results are as follows.
Accuracy of K-means method: 0.866Accuracy of Convex Hull method: 0.867
There is a slight improvement in my convex hull method over K-means.
The article titled High-dimensional data clustering by using local affine/convex hulls by HakanCevikalp shows that the convex hull-based method they proposed avoids the “hole artefacts” problem (the sparse and irregular distributions in high-dimensional spaces can make the nearest-neighbour distances unreliable) and improves the accuracy of high-dimensional datasets over other state-of-the-art subspace clustering methods.
www.sciencedirect.com
You can find the jupyter notebook containing the code used for this article.
Hope this article was interesting and useful.