A Dive Into Unsupervised Learning
By: Abhijit Roy
Tags: Unsupervised Learning, Hierarchical Clustering, K Means Clustering

The first image shows the distribution of points on which the clustering is to be performed. In the second, we just start clustering randomly and we locate their centroids. Now, we calculate the SSE for each cluster, i.e, the sum of squared distances of all the points in the cluster from the centroid of the cluster. The main aim is to keep decreasing the SSE as we move forward. Coming to the last image or fourth image we have reached the optimal clusters and no the centroids do not shift anymore. Here we will find the SSE to be minimum.

We find the total SSE for a stage in the clustering process by adding up the sum of all individual clusters. So, if a point of cluster A is closer to the centroid of cluster B than that of A, grouping the point in cluster B will decrease the overall SSE.

We have seen that for this type of clustering technique we need a user-defined parameter ‘K’ which defines the number of clusters that need to be created. Now, this is a very important parameter. To, find this parameter a number of methods are used. The most important and used method is the elbow method.

For smaller datasets, k=(N/2)^(1/2) or the square root of half of the number of points in the distribution.

Elbow method

The elbow method works on the Sum of the squared error principle. We try random numbers of clusters and try to build a graph of the correspondingly obtained SSEs. So, say for our case, we consider the number of clusters as 1,2,3,4,5,6,7, and 8. For each number of clusters we obtained the corresponding SSEs, SSE1, SSE2, SSE3, and so on. Now, if we plot the SSEs we obtain a plot as follows:

Image by author

So, we find an Elbow at 4, so, we should use 4 as the optimal number of clusters. On moving further also, the SSE decreases, but it creates a huge number of clusters which is not desired. This is a trade-off between the decreasing SSE and the number of created clusters.

Disadvantages

One of the main disadvantages of the algorithm is that it is superbly susceptible to noise and outliers. As we have seen the algorithm selects the centroid which is just calculated as the mean of all the points, and not a real point of the distribution, the outliers present in any cluster will cause the centroid to distort and it will also cause the SSE to blow up.

To solve this issue with the K-means algorithm, the K-medoid algorithm was developed.

K-medoid Algorithm

Instead of considering the mean of the data points in the cluster, the K-medoids technique considers ‘k’ representative data points from the existing pointed in the data sets as the center of the cluster. The representatives are actual data points in K-medoids, unlike K-means algorithms. Next, the process is very similar to the K-means algorithm. In this case, we calculate the SSE from the considered representative point and not the centroid as we did in the case of K-means.

An algorithm named Partitioning Around Medoids (PAM) is used to implement the K-medoid algorithm. Let’s see how the algorithm works.

Initially, K points are chosen randomly as the representative points, one for each cluster. We assign the remaining points to the cluster to whose representative point it is nearest to. We randomly select a non-representative point say ‘On’ from a cluster whose current representative point is ‘Oo’. So, we calculate both the ‘SSEold’ and ‘SSEnew’ corresponding to the ‘Oo’ and ‘On’ points. If we find SSEnew<SSEold we swap the representative points and redefine the clusters We repeat 2 to 5 until there is no change in SSE.

The algorithm works on real points rather than imaginary mean so, they can easily eliminate outliers and noise. The disadvantage of the algorithm is that it has higher time complexity. The time complexity of K-medoid algorithm O(k(n-k)²) for one iteration where k is the number of representative points and n is the total number of points in the distribution.

So, for the total algorithm to complete, if we take t iterations the time complexity will be O(tk(n-k)²)

K-NN Algorithm

K-NN or K-Nearest Neighbours do not have a model-based working. It is a simple algorithm that works entirely on similarity metrics that is mostly distance-based measure. It considers all the points available and finds the distance, so the K-NN algorithm has a huge time complexity and only used for small datasets.

We have seen the partitioning methods, now let’s look at the hierarchical methods.

Hierarchical methods

There are two types of hierarchical methods:

Agglomerative Hierarchical clustering: In this type of hierarchical clustering, each point initially starts as a cluster, and slowly the nearest or similar most clusters merge to create one cluster. Divisive Hierarchical Clustering: The type of hierarchical clustering is just the opposite of Agglomerative clustering. In this type, all the points start as one large cluster and slowly the clusters get divided into smaller clusters based on how large the distance or less similarity is between the two clusters. We keep on dividing the clusters until all the points become individual clusters.

Now, as we have seen in both cases distance-based similarity metrics play a major role. One thing to note is, though we are using the term “Distance-based Similarity” it does not mean only distance, it implies measures derived from a distance as basic units. We will look at them later.

For agglomerative clustering, we keep on merging the clusters which are nearest or have a high similarity score to one cluster. So, if we define a cut-off or threshold score for the merging we will get multiple clusters instead of a single one. For instance, if we say the threshold similarity metrics score is 0.5, it means the algorithm will stop merging the clusters if no two clusters are found with a similarity score less than 0.5, and the number of clusters present at that step will give the final number of clusters that need to be created to the clusters.

Similarly, for divisive clustering, we divide the clusters based on the least similarity scores. So, if we define a score of 0.5, it will stop dividing or splitting if the similarity score between two clusters is less than or equal to 0.5. We will be left with a number of clusters and it won’t reduce to every point of the distribution.

Let’s see a visualization for agglomerative hierarchical clustering.