K-Means vs. DBSCAN Clustering
By: Ekta Sharma
Tags: Dbscan, Elbow Method, Clustering, K Means, Epsilon

Photo Credit — Unsplash

Clustering — An Unsupervised Machine Learning Technique

Clustering is grouping of unlabeled data points in such a way that:

The data points within the same group are similar to each other, and

the data points in different groups are dissimilar to each other.

The goal is to create clusters that have high intra-cluster similarity and low inter-cluster similarity.

K-Means Cluster

K-Means cluster is one of the most commonly used unsupervised machine learning clustering techniques. It is a centroid based clustering technique that needs you decide the number of clusters (centroids) and randomly places the cluster centroids to begin the clustering process. The goal is to divide N observations into K clusters repeatedly until no more groups can be formed.

Advantages of K-Means

Easy to understand and implement.

Can handle large datasets well.

Disadvantages of K-Means

Sensitive to number of clusters/centroids chosen. Even after using techniques like Elbow method, it is sometimes hard to generate good clusters.

Does not work well with outliers. Centroids can get dragged by the outliers resulting in skewed clusters.

Gets difficult in high dimensional spaces as the distance between the points increases and Euclidean distance diverges (converges to a constant value).

Gets slow as the number of dimensions increases.

K-Means Algorithm

1. Decide the number of clusters. This number is called K and number of clusters is equal to the number of centroids. Based on the value of K, generate the coordinates for K random centroids.

2. For every point, calculate the Euclidean distance between the point and each of the centroids.

3. Assign the point to its nearest centroid. The points assigned to the same centroid form a cluster.

4. Once clusters are formed, calculate new centroid for each cluster by taking the cluster mean. Cluster mean is the mean of the x and y coordinates of all the points belonging to the cluster.

5. Repeat step 2, 3 and 4 until the centroids cannot move any further. In other words, repeat these steps until convergence.

K-Means for the Iris Dataset using Scikit Learn

import pandas as pd

from sklearn import metrics

from sklearn.cluster import KMeans

import matplotlib.pyplot as plt # reading the classic iris dataset into a df

iris_df = pd.read_csv(“iris_dataset.csv”) # Setting the independent features (input)

X = iris_df.drop(“species”, axis=1).values # Creating the KMeans object and fitting it to the Iris data

iris_kmeans = KMeans(n_clusters=3) iris_kmeans.fit(X) # Predicting the cluster labels

labels = iris_kmeans.predict(X)

print(labels) # Finding the final centroids

centroids = iris_kmeans.cluster_centers_ # Evaluating the quality of clusters

s = metrics.silhouette_score(X, labels, metric=’euclidean’)

print(f”Silhouette Coefficient for the Iris Dataset Clusters: {s:.2f}”) # plotting the clusters using sepal_length and sepal_width

plt.scatter(X[:, 0], X[:, 1], c=labels, cmap=”rainbow”)

plt.show()

Output

Label values represent the cluster number.

[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 2 2 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 0 2 0 0 0 0 2 0 0 0 0

0 0 2 2 0 0 0 0 2 0 2 0 2 0 0 2 2 0 0 0 0 0 2 0 0 0 0 2 0 0 0 2 0 0 0 2 0

0 2] Silhouette Coefficient for the Iris Dataset Clusters: 0.55

Three K-Means Clusters

Finding optimal number of Clusters for K-Means (Elbow Method)

The quality of clusters formed using K-Means largely depends on the selected value of K. A wrong choice of K can lead to poor clustering. So how to select K? Let’s take a look at the commonly used technique called “Elbow Method”. The goal is to select the K at which an elbow is formed.

Steps:

1. For different values of K, execute the following steps:

2. For each cluster, calculate the sum of squared distance of every point to its centroid.

3. Add the sum of squared distances of each cluster to get the total sum of squared distances for that value of K.

4. Keep adding the total sum of squared distances for each K to a list.

5. Plot the sum of squared distances (using the list created in the previous step) and their K values.

6. Select the K at which a sharp change occurs (looks like an elbow of the curve).

# Arbitrarily selecting a range of values for K

K = range(1,10) sum_of_squared_distances = [] # Using Scikit Learn’s KMeans Algorithm to find sum of squared distances for k in K:

model = KMeans(n_clusters=k).fit(X)

sum_of_squared_distances.append(model.inertia_) plt.plot(K, sum_of_squared_distances, “bx-”)

plt.xlabel(“K values”)

plt.ylabel(“Sum of Squared Distances”)

plt.title(“Elbow Method”)

plt.show()

Looking at this plot, we can see the elbow at K=3 and hence that is our optimal number of clusters for this dataset.

Sometimes we can end up with multiple values showing an elbow. In this case, to find the best K, an evaluation metric like Silhouette Coefficient can be used. The K that will return the highest positive value for the Silhouette Coefficient should be selected.