K-Means vs. DBSCAN Clustering — For Beginners | by Ekta Sharma | Towards Data Science

Clustering is grouping of unlabeled data points in such a way that: The data points within the same group are similar to each other, and the data points in different groups are dissimilar to each other.The goal is to create clusters that have high intra-cluster similarity and low inter-cluster similarity.
K-Means cluster is one of the most commonly used unsupervised machine learning clustering techniques. It is a centroid based clustering technique that needs you decide the number of clusters (centroids) and randomly places the cluster centroids to begin the clustering process. The goal is to divide N observations into K clusters repeatedly until no more groups can be formed.
Easy to understand and implement.
Can handle large datasets well.
Sensitive to number of clusters/centroids chosen. Even after using techniques like Elbow method, it is sometimes hard to generate good clusters.
Does not work well with outliers. Centroids can get dragged by the outliers resulting in skewed clusters.
Gets difficult in high dimensional spaces as the distance between the points increases and Euclidean distance diverges (converges to a constant value).
Gets slow as the number of dimensions increases.
1. Decide the number of clusters. This number is called K and number of clusters is equal to the number of centroids. Based on the value of K, generate the coordinates for K random centroids.
2. For every point, calculate the Euclidean distance between the point and each of the centroids.
3. Assign the point to its nearest centroid. The points assigned to the same centroid form a cluster.
4. Once clusters are formed, calculate new centroid for each cluster by taking the cluster mean. Cluster mean is the mean of the x and y coordinates of all the points belonging to the cluster.
5. Repeat step 2, 3 and 4 until the centroids cannot move any further. In other words, repeat these steps until convergence.
import pandas as pdfrom sklearn import metricsfrom sklearn.cluster import KMeansimport matplotlib.pyplot as plt# reading the classic iris dataset into a dfiris_df = pd.read_csv(“iris_dataset.csv”)# Setting the independent features (input)X = iris_df.drop(“species”, axis=1).values# Creating the KMeans object and fitting it to the Iris datairis_kmeans = KMeans(n_clusters=3)iris_kmeans.fit(X)# Predicting the cluster labelslabels = iris_kmeans.predict(X)print(labels)# Finding the final centroidscentroids = iris_kmeans.cluster_centers_# Evaluating the quality of clusterss = metrics.silhouette_score(X, labels, metric=’euclidean’)print(f”Silhouette Coefficient for the Iris Dataset Clusters: {s:.2f}”)# plotting the clusters using sepal_length and sepal_widthplt.scatter(X[:, 0], X[:, 1], c=labels, cmap=”rainbow”)plt.show()
Label values represent the cluster number.
[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 11 1 1 1 1 1 1 1 1 1 1 1 1 2 2 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 22 2 2 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 0 2 0 0 0 0 2 0 0 0 00 0 2 2 0 0 0 0 2 0 2 0 2 0 0 2 2 0 0 0 0 0 2 0 0 0 0 2 0 0 0 2 0 0 0 2 00 2]Silhouette Coefficient for the Iris Dataset Clusters: 0.55
The quality of clusters formed using K-Means largely depends on the selected value of K. A wrong choice of K can lead to poor clustering. So how to select K? Let’s take a look at the commonly used technique called “Elbow Method”. The goal is to select the K at which an elbow is formed.
Steps:
1. For different values of K, execute the following steps:
2. For each cluster, calculate the sum of squared distance of every point to its centroid.
3. Add the sum of squared distances of each cluster to get the total sum of squared distances for that value of K.
4. Keep adding the total sum of squared distances for each K to a list.
5. Plot the sum of squared distances (using the list created in the previous step) and their K values.
6. Select the K at which a sharp change occurs (looks like an elbow of the curve).
# Arbitrarily selecting a range of values for KK = range(1,10)sum_of_squared_distances = []# Using Scikit Learn’s KMeans Algorithm to find sum of squared distancesfor k in K:    model = KMeans(n_clusters=k).fit(X)    sum_of_squared_distances.append(model.inertia_)plt.plot(K, sum_of_squared_distances, “bx-”)plt.xlabel(“K values”)plt.ylabel(“Sum of Squared Distances”)plt.title(“Elbow Method”)plt.show()
Looking at this plot, we can see the elbow at K=3 and hence that is our optimal number of clusters for this dataset.
Sometimes we can end up with multiple values showing an elbow. In this case, to find the best K, an evaluation metric like Silhouette Coefficient can be used. The K that will return the highest positive value for the Silhouette Coefficient should be selected.
DBSCAN is a density-based clustering algorithm that forms clusters of dense regions of data points ignoring the low-density areas (considering them as noise).
Works well for noisy datasets.
Can identity Outliers easily.
Clusters can take any irregular shape unlike K-Means where clusters are more or less spherical.
Does not work very well for sparse datasets or datasets with varying density.
Sensitive to eps and minPts parameters.
Not partitionable for multiprocessor systems.
DBSCAN uses the following two user defined parameters for clustering:
Epsilon (eps): It is defined as the maximum distance between two points to be considered as neighboring points (belonging to the same cluster).
Minimum Points (min_samples or minPts): This defines the minimum number of neighboring points that a given point needs to be considered a core data point. This includes the point itself. For example, if minimum number of points is set to 4, then a given point needs to have 3 or more neighboring data points to be considered a core data point.
If minimum number of points meet the epsilon distance requirement then they are considered as a cluster.
Core Point: A data point is considered to be a core point if it has minimum number of neighboring data points (min_samples) at an epsilon distance from it. This minimum number of data points includes the original data point.Border Point: A data point that has less than minimum number of data points needed but has at least one core point in the neighborhood.Noise: A data point that is not a core point or a border point is considered noise or an outlier.
1. Decide the value of eps and minPts.
2. For each point:
Calculate its distance from all other points. If the distance is less than or equal to eps then mark that point as a neighbor of x.
If the point gets a neighboring count greater than or equal to minPts, then mark it as a core point or visited.
3. For each core point, if it not already assigned to a cluster than create a new cluster. Recursively find all its neighboring points and assign them the same cluster as the core point.
4. Continue these steps until all the unvisited points are covered.
import pandas as pdfrom sklearn import metricsfrom sklearn.cluster import DBSCANimport matplotlib.pyplot as plt# reading the classic iris dataset into a dfiris_df = pd.read_csv(“iris_dataset.csv”)X = iris_df.drop(“species”, axis=1).valuesiris_dbscan = DBSCAN(eps=0.5, min_samples=5)iris_dbscan.fit(X)labels = iris_dbscan.labels_# label=-1 means the point is an outlier. Rest of the values represent the label/cluster number starting from 0print(labels)# Creating a numpy array with all values set to false by defaultcore_samples_mask = np.zeros_like(labels, dtype=bool)# Setting core and border points (all points that are not -1) to Truecore_samples_mask[iris_dbscan.core_sample_indices_] = True# Finding the number of clusters in labels (ignoring noise if present)n_clusters_ = len(set(labels)) — (1 if -1 in labels else 0)n_noise_ = list(labels).count(-1)# Printing the number of clusters and number of noise points (outliers)print(“Estimated number of clusters: %d” % n_clusters_)print(“Estimated number of noise points: %d” % n_noise_)# Evaluating the quality of clusterss = metrics.silhouette_score(X, iris_dbscan.labels_)print(f”Silhouette Coefficient for the Iris Dataset Clusters: {s:.2f}”)
Label = -1 means it is a noise point (outlier).Label = 0 or more, indicates the cluster number.
[ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -1 0 0 0 0 0 00 0 1 1 1 1 1 1 1 -1 1 1 -1 1 1 1 1 1 1 1 -1 1 1 11 1 1 1 1 1 1 1 1 1 1 1 1 1 1 -1 1 1 1 1 1 -1 1 11 1 -1 1 1 1 1 1 1 -1 -1 1 -1 -1 1 1 1 1 1 1 1 -1 -1 11 1 -1 1 1 1 1 1 1 1 1 -1 1 1 -1 -1 1 1 1 1 1 1 1 11 1 1 1 1 1]Estimated number of clusters: 2Estimated number of noise points: 17Silhouette Coefficient for the Iris Dataset Clusters: 0.49
unique_labels = set(labels)colors = plt.cm.Spectral(np.linspace(0, 1, len(unique_labels)))for k, col in zip(unique_labels, colors):    if k == -1:        # Black used for noise        col = “k”        class_member_mask = (labels == k)    xy = X[class_member_mask & core_samples_mask]    plt.plot(xy[:, 0], xy[:, 1], “o”, markerfacecolor=col,    markeredgecolor=”k”, markersize=10)    xy = X[class_member_mask & ~core_samples_mask]    plt.plot(xy[:, 0], xy[:, 1], “o”, markerfacecolor=col,    markeredgecolor=”k”, markersize=5)plt.title(“Estimated number of clusters: %d” % n_clusters_)plt.show()
DBSCAN clustering algorithm is sensitive to the eps value we choose. So how can we know that we have selected the optimal eps value? Here is a commonly used technique called “Knee method”. The goal is to find the average of distances for every point to its K nearest neighbors and select the distance at which maximum curvature or a sharp change happens. The value of K is set to be equal to minPoints.
Here is an example to show optimal eps value selection using Scikit Learn’s NearestNeighbors module.
from sklearn.neighbors import NearestNeighborsnearest_neighbors = NearestNeighbors(n_neighbors=5)nearest_neighbors.fit(X)distances, indices = nearest_neighbors.kneighbors(X)distances = np.sort(distances, axis=0)[:, 1]print(distances)plt.plot(distances)plt.show()
[0. 0. 0. 0. 0. 0.10.1 0.1 0.1 0.1 0.1 0.10.1 0.14142136 0.14142136 0.14142136 0.14142136 0.141421360.14142136 0.14142136 0.14142136 0.14142136 0.14142136 0.141421360.14142136 0.14142136 0.14142136 0.14142136 0.14142136 0.141421360.14142136 0.14142136 0.14142136 0.14142136 0.14142136 0.141421360.14142136 0.14142136 0.14142136 0.14142136 0.14142136 0.141421360.14142136 0.14142136 0.14142136 0.14142136 0.14142136 0.141421360.17320508 0.17320508 0.17320508 0.17320508 0.17320508 0.173205080.17320508 0.2 0.2 0.2 0.2 0.20.2 0.2 0.2 0.2 0.2 0.20.2236068 0.2236068 0.2236068 0.2236068 0.2236068 0.22360680.2236068 0.2236068 0.2236068 0.2236068 0.24494897 0.244948970.24494897 0.24494897 0.24494897 0.24494897 0.24494897 0.244948970.24494897 0.24494897 0.24494897 0.24494897 0.26457513 0.264575130.26457513 0.26457513 0.26457513 0.26457513 0.26457513 0.264575130.26457513 0.26457513 0.26457513 0.26457513 0.28284271 0.282842710.28284271 0.28284271 0.3 0.3 0.3 0.30.3 0.3 0.3 0.31622777 0.31622777 0.316227770.33166248 0.33166248 0.33166248 0.33166248 0.33166248 0.346410160.34641016 0.34641016 0.34641016 0.34641016 0.34641016 0.346410160.36055513 0.36055513 0.36055513 0.36055513 0.37416574 0.387298330.38729833 0.38729833 0.41231056 0.41231056 0.41231056 0.412310560.42426407 0.42426407 0.43588989 0.45825757 0.48989795 0.489897950.53851648 0.53851648 0.55677644 0.6244998 0.63245553 0.73484692]
The optimal value should be the value at which we see maximum curvature which in this case seems to be near 0.5.
Sometimes we can end up with multiple values showing a sharp change. In this case, to find the best K, an evaluation metric like Silhouette Coefficient can be used. The K that will return the highest positive value for the Silhouette Coefficient should be selected.
When to use which of these two clustering techniques, depends on the problem. Even though K-Means is the most popular clustering technique, there are use cases where using DBSCAN results in better clusters.