Explaining DBSCAN Clustering. Using DBSCAN to identify employee... | by Kamil Mysiak | Towards Data Science

Density-based spatial clustering of applications with noise (DBSCAN) is an unsupervised clustering ML algorithm. Unsupervised in the sense that it does not use pre-labeled targets to cluster the data points. Clustering in the sense that it attempts to group similar data points into artificial groups or clusters. It is an alternative to popular clustering algorithms such as KMeans and hierarchical clustering.
In our example, we will be examining a human resources dataset consisting of 15,000 individual employees. The dataset contains employee job characteristics such as job satisfaction, performance score, workload, years of tenure, accidents, number of promotions.
In my previous article, we examined the KMeans clustering algorithm which I strongly suggest you read (LINK). You will be able to orient yourself to how KMeans clusters the data.
KMeans is especially vulnerable to outliers. As the algorithm iterates through centroids, outliers have a significant impact on the way the centroids moves before reaching stability and convergence. Furthermore, KMeans has problems accurately clustering data where the clusters are of different sizes and densities. K-Means can only apply spherical clusters and its accuracy will suffer if the data is not spherical. Last but not least, KMeans requires us to first select the number of clusters we wish to find. Below is an example of how KMeans and DBSCAN would individually cluster the same dataset.
On the other hand, DBSCAN does not require us to specify the number of clusters, avoids outliers, and works quite well with arbitrarily shaped and sized clusters. It does not have centroids, the clusters are formed by a process of linking neighbor points together.
First, let’s define Epsilon and Minimum Points, two required parameters when applying the DBSCAN algorithm, and some additional terminology.
Epsilon (ɛ): Max radius of the neighborhood. Data points will be valid neighbors if their mutual distance is less than or equal to the specified epsilon. In other words, it is the distance that DBSCAN uses to determine if two points are similar and belong together. A larger epsilon will produce broader clusters (encompassing more data points) and a smaller epsilon will build smaller clusters. In general, we prefer smaller values because we only want a small fraction of data points within the epsilon distance from each other. Too small and you’ll begin to split valid clusters into smaller and smaller clusters. Too big and you’ll aggregate valid clusters into 2–3 massive clusters with little business insights.Minimum Points (minPts): The minimum number of data points within the radius of a neighborhood (ie. epsilon) for the neighborhood to be considered a cluster. Keep in mind that the initial point is included in the minPts. A low minPts help the algorithm build more clusters with more noise or outliers. A higher minPts will ensure more robust clusters but if it’s too large smaller clusters will be incorporated into larger clusters.
Epsilon (ɛ): Max radius of the neighborhood. Data points will be valid neighbors if their mutual distance is less than or equal to the specified epsilon. In other words, it is the distance that DBSCAN uses to determine if two points are similar and belong together. A larger epsilon will produce broader clusters (encompassing more data points) and a smaller epsilon will build smaller clusters. In general, we prefer smaller values because we only want a small fraction of data points within the epsilon distance from each other. Too small and you’ll begin to split valid clusters into smaller and smaller clusters. Too big and you’ll aggregate valid clusters into 2–3 massive clusters with little business insights.
Minimum Points (minPts): The minimum number of data points within the radius of a neighborhood (ie. epsilon) for the neighborhood to be considered a cluster. Keep in mind that the initial point is included in the minPts. A low minPts help the algorithm build more clusters with more noise or outliers. A higher minPts will ensure more robust clusters but if it’s too large smaller clusters will be incorporated into larger clusters.
If “minimum points” = 4, any 4 or more points within the epsilon distance away from each other will be considered a cluster.
Core Points: Core data points have at least minPts number of data points within their epsilon distance.
I had always believed DBSCAN needed a third parameter named “core_min” which would determine the minimum number of core points before a cluster of neighborhood points can be considered a valid cluster.
Border Points: Border data points are on the outskirts as they are in the neighborhood (ie. w/in epsilon distance of core point) but have less than the required minPts.
Outlier Points: These points are not part of a neighborhood (ie. more than epsilon distance) and are not border points. These are points located in low-density areas.
First, a random point is selected which has at least minPts within its epsilon radius. Then each point that is within the neighborhood of the core point is evaluated to determine if it has the minPts nearby within the epsilon distance (minPts includes the point itself). If the point does meet the minPts criteria it becomes another core point and the cluster expands. If a point does not meet the minPts criteria it becomes a border point. As the process continues a chain begins to develop as core point “a” is a neighbor of “b” which is a neighbor or “c” and so on. The cluster is complete as it becomes surrounded by border points because there are no more points within the epsilon distance. A new random point is selected and the process repeats to identify the next cluster.
One method used to estimate the optimal epsilon value is to use nearest neighbor distances. If you recall, nearest neighbors is a supervised ML clustering algorithm which clusters new data points based on their distance from other “known” data points. We train a KNN model on labeled training data to determine which data points belong to which cluster. Then when we apply the model to new data, the algorithm determines which cluster the new data point belongs to based on the distance to trained clusters. We do have to determine the “k” parameter a-priori which specifies how many closest or nearest neighboring points the model will consider before assigning the new data point to a cluster.
To determine the best epsilon value, we calculate the average distance between each point and its closest/nearest neighbors. We then plot a k-distance and choose the epsilon value at the “elbow” of the graph. On the y-axis, we plot the average distances and the x-axis all the data points in your dataset.
if epsilon is chosen much too small, a large part of the data will not be clustered, whereas a high epsilon value clusters will merge and the majority of data points will be in the same cluster. In general, small values of epsilon are preferable, and as a rule of thumb, only a small fraction of points should be within this distance of each other.
Typically, we should set minPts to be greater or equal to the number of dimensionality of our dataset. That said, we often see folks multiplying the number of features X 2 to determine their minPts value.
Much like the “Elbow Method” used to determine the optimal epsilon value the minPts heuristic isn’t correct 100% of the time.
Silhouette Method: This technique measures the separability between clusters. First, an average distance is found between each point and all other points in a cluster. Then it measures the distance between each point and each point in other clusters. We subtract the two average measures and divide by whichever average is larger.
We ultimately want a high (ie. closest to 1) score which would indicate that there is a small intra-cluster average distance (tight clusters) and a large inter-cluster average distance (clusters well separated).
Visual Cluster Interpretation: Once you have obtained your clusters it is very important to interpret each cluster. This is typically done by merging the original dataset with the clusters and visualizing each cluster. The more clear and distinct each cluster is the better. We will review this process below.
Does not require us to pre-determine the number of clusters like KMeans
Handles outliers like a champ
Can separate high density data into small clusters
Can cluster non-linear relationships (finds arbitrary shapes)
Struggles to identify clusters within data of varying density
Can suffer with high dimensional data
Very sensitive to epsilon and minimum points parameters
import numpy as npimport pandas as pdimport matplotlib.pyplot as pltimport seaborn as snsimport plotly.offline as pyopyo.init_notebook_mode()import plotly.graph_objs as gofrom plotly import toolsfrom plotly.subplots import make_subplotsimport plotly.offline as pyimport plotly.express as pxfrom sklearn.cluster import DBSCANfrom sklearn.neighbors import NearestNeighborsfrom sklearn.metrics import silhouette_scorefrom sklearn.preprocessing import StandardScalerfrom sklearn.decomposition import PCAplt.style.use('fivethirtyeight')from warnings import filterwarningsfilterwarnings('ignore')with open('HR_data.csv') as f:    df =  pd.read_csv(f, usecols=['satisfaction_level', 'last_evaluation', 'number_project',       'average_montly_hours', 'time_spend_company', 'Work_accident',       'promotion_last_5years'])f.close()
Since the features in our dataset are not on the same scale we need to standardize the entire dataset. In other words, each feature in our dataset has unique magnitudes and range for their data. A one-point increase in Satisfaction_level does not equal a one-point increase in Last_evaluation and vice versa. Since DBSCAN utilizes the distance (Euclidean) between points to determine similarity, unscaled data creates a problem. If one feature has higher variability in its data the distance calculation will be more affected by that feature. By scaling our features we align all the features to a mean of zero and a standard deviation of one.
scaler = StandardScaler()scaler.fit(df)X_scale = scaler.transform(df)df_scale = pd.DataFrame(X_scale, columns=df.columns)df_scale.head()
Some algorithms such as KMeans find it difficult to accurately construct clusters if the dataset has too many features (ie. high dimensionality). High dimensionality does not necessarily mean hundreds or even thousands of features. Even 10 features can create accuracy issues.
The theory behind feature or dimensionality reduction is to convert the original feature set into fewer artificially derived features which still maintain most of the information encompassed in the original features.
One of the most prevalent feature reduction techniques is Principal Component Analysis or PCA. PCA reduces the original dataset into a specified number of features which PCA calls principal components. We have to select the number of principal components we wish to see. We discuss feature reduction in my article on KMeans clustering and I strongly advise you to take a look (LINK).
First, we need to determine the appropriate number of principal components. It would seem that 3 principal components account for roughly 75% of the variance.
pca = PCA(n_components=7)pca.fit(df_scale)variance = pca.explained_variance_ratio_ var=np.cumsum(np.round(variance, 3)*100)plt.figure(figsize=(12,6))plt.ylabel('% Variance Explained')plt.xlabel('# of Features')plt.title('PCA Analysis')plt.ylim(0,100.5)plt.plot(var)
Now that we know the number of principal components needed to maintain a specific percentage of variance let’s apply a 3-component PCA to our original dataset. Notice that the first principal component accounts for 26% of the variance from the original dataset. We will utilize the “pca_df” data frame for the remainder of this article.
pca = PCA(n_components=3)pca.fit(df_scale)pca_scale = pca.transform(df_scale)pca_df = pd.DataFrame(pca_scale, columns=['pc1', 'pc2', 'pc3'])print(pca.explained_variance_ratio_)
Plotting our data in a 3D space we can see some potential problems for DBSCAN. If you recall one of the main cons for DBSCAN is its inability to accurately cluster data of varying density and from the plot below we can see two separate clusters of very different density. Upon applying the DBSCAN algorithm we might be able to find clusters in the lower cluster of data points but many of the data points in the upper cluster might be classified as outliers/noise. This is of course all dependent on our selection of epsilon and minimum point values.
Scene = dict(xaxis = dict(title  = 'PC1'),yaxis = dict(title  = 'PC2'),zaxis = dict(title  = 'PC3'))trace = go.Scatter3d(x=pca_df.iloc[:,0], y=pca_df.iloc[:,1], z=pca_df.iloc[:,2], mode='markers',marker=dict(colorscale='Greys', opacity=0.3, size = 10, ))layout = go.Layout(margin=dict(l=0,r=0),scene = Scene, height = 1000,width = 1000)data = [trace]fig = go.Figure(data = data, layout = layout)fig.show()
Approach #1
Before we apply the clustering algorithm we have to determine the appropriate epsilon level using the “Elbow Method” we discussed above. It would seem the optimal epsilon value is around 0.2. Finally, since we have 3 principal components to our data we’ll set our minimum points criteria to 6.
plt.figure(figsize=(10,5))nn = NearestNeighbors(n_neighbors=5).fit(pca_df)distances, idx = nn.kneighbors(pca_df)distances = np.sort(distances, axis=0)distances = distances[:,1]plt.plot(distances)plt.show()
Setting the epsilon to 0.2 and min_samples to 6 has resulted in 53 clusters, a Silhouette score of -0.521, and over 1500 data points which are considered outliers/noise. There may be some research areas where 53 clusters might be considered informative but we have a dataset of 15,000 employees. From a business perspective, we need a manageable number of clusters (ie 3–5) which can be used to better understand the workplace. Also, a silhouette score of -0.521 would indicate data points are incorrectly clustered.
Looking at the 3D plot below we can see one cluster encompassing the majority of the data points. There is a smaller but significant cluster which emerged but that leaves 52 remaining clusters of much smaller size. From a business perspective these clusters are not very informative as most employees belong to just two clusters. The organization would want to see a few large clusters in order to be certain of their validity but also be able to engage in some organization initiatives against the clustered employees (ie. increased training, compensation changes, etc.).
db = DBSCAN(eps=0.2, min_samples=6).fit(pca_df)labels = db.labels_# Number of clusters in labels, ignoring noise if present.n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)n_noise_ = list(labels).count(-1)print('Estimated number of clusters: %d' % n_clusters_)print('Estimated number of noise points: %d' % n_noise_)print("Silhouette Coefficient: %0.3f" % metrics.silhouette_score(pca_df, labels))
Scene = dict(xaxis = dict(title  = 'PC1'),yaxis = dict(title  = 'PC2'),zaxis = dict(title  = 'PC3'))labels = db.labels_trace = go.Scatter3d(x=pca_df.iloc[:,0], y=pca_df.iloc[:,1], z=pca_df.iloc[:,2], mode='markers',marker=dict(color = labels, colorscale='Viridis', size = 10, line = dict(color = 'gray',width = 5)))layout = go.Layout(scene = Scene, height = 1000,width = 1000)data = [trace]fig = go.Figure(data = data, layout = layout)fig.update_layout(title='DBSCAN clusters (53) Derived from PCA', font=dict(size=12,))fig.show()
Approach #2
Instead of using the “Elbow Method” and the minimum value heuristic let’s take an iterative approach to fine-tuning our DBSCAN model. We are going to iterate through a range of epsilon and minimum point values as we apply the DBSCAN algorithm to our data.
In our example, we are going to iterative through epsilon values ranging from 0.5 to 1.5 at 0.1 intervals and minimum point values ranging from 2–7. The for-loop will run the DBSCAN algorithm using the set of values and produce the number of clusters and silhouette score for each iteration. Keep in mind you will need to adjust your parameters according to your data. You might run this code on one set of parameters and find that the best silhouette score produced is 0.30. You might want to increase the epsilon value in order to encompass more points into a cluster.
pca_eps_values = np.arange(0.2,1.5,0.1) pca_min_samples = np.arange(2,5) pca_dbscan_params = list(product(pca_eps_values, pca_min_samples))pca_no_of_clusters = []pca_sil_score = []pca_epsvalues = []pca_min_samp = []for p in pca_dbscan_params:    pca_dbscan_cluster = DBSCAN(eps=p[0], min_samples=p[1]).fit(pca_df)    pca_epsvalues.append(p[0])    pca_min_samp.append(p[1])pca_no_of_clusters.append(len(np.unique(pca_dbscan_cluster.labels_)))    pca_sil_score.append(silhouette_score(pca_df, pca_dbscan_cluster.labels_))pca_eps_min = list(zip(pca_no_of_clusters, pca_sil_score, pca_epsvalues, pca_min_samp))pca_eps_min_df = pd.DataFrame(pca_eps_min, columns=['no_of_clusters', 'silhouette_score', 'epsilon_values', 'minimum_points'])pca_ep_min_df
We can see that iterating through our epsilon and minimum values we had obtained a wide range of number of clusters and silhouette scores. Epsilon scores between 0.9 and 1.1 began to produce a manageable number of clusters. Increasing epsilon to 1.2 and above creates too few clusters to make much business sense. Furthermore, some of those clusters may be just noise (ie. -1) which we’ll get to in a bit.
It is also important to understand that increasing epsilon decreases the number of clusters but each cluster will also begin to encompass more outlier/noise data points. There is a certain level of diminishing returns.
For the sake of simplicity let’s choose 7 clusters and examine what the cluster distribution looks like. (epsilon: 1.0 & minPts: 4).
It is also important to point out a frequent error you will surely encounter running this string of code. Sometimes when you have set your parameters (ie. eps_values and min_samples) too broad the for-loop will eventually come to a combination of eps_values and min_samples which only produces one cluster. However, the Silhouette_score function requires at least two clusters to be defined. You will need to restrict your parameters to avoid this issue.
In our example above, if we had set our epsilon parameter range from 0.2 to 2.5 that would most likely result in one cluster and ultimately elicit the error.
You might be asking yourself “weren’t we supposed to obtain 7 clusters?”. The answer is “Yes” and if we look at the unique labels/clusters we see 7 labels for each data point. Per Sklearn documentation, a label of “-1” equates to a “noisy” data point it hasn’t been clustered into any of the 6 high-density clusters. We naturally don’t want to consider any labels of “-1” as a cluster, therefore, they are removed from the calculation.
db = DBSCAN(eps=1.0, min_samples=4).fit(pca_df)labels = db.labels_# Number of clusters in labels, ignoring noise if present.n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)n_noise_ = list(labels).count(-1)print('Estimated number of clusters: %d' % n_clusters_)print('Estimated number of noise points: %d' % n_noise_)print("Silhouette Coefficient: %0.3f" % silhouette_score(pca_df, labels))
set(labels)
Looking at the 3D plot of the six DBSCAN derived clusters it seems the less dense cluster towards the top of the plot did not pose much of a problem for DBSCAN despite it being less dense. If you recall, DBSCAN has a difficult time properly clustering data of various density. The distance between the top cluster and the much larger bottom cluster was most likely larger than our epsilon value of 1.0.
That said, the dataset contains additional high-density clusters but our epsilon and minimum values are too large. The bottom cluster contains at least two high-density clusters, however, due to the strong density of the bottom cluster lowering the epsilon and minimum point values would simply create many smaller clusters. Once again this is the major downside of DBSCAN. I always believed DBSCAN needed a 3rd parameter, “min_core”, which would determine the minimum number of core points before a cluster can be considered a valid cluster.
Scene = dict(xaxis = dict(title  = 'PC1'),yaxis = dict(title  = 'PC2'),zaxis = dict(title  = 'PC3'))# model.labels_ is nothing but the predicted clusters i.e y_clusterslabels = db.labels_trace = go.Scatter3d(x=pca_df.iloc[:,0], y=pca_df.iloc[:,1], z=pca_df.iloc[:,2], mode='markers',marker=dict(color = labels, colorscale='Viridis', size = 10, line = dict(color = 'gray',width = 5)))layout = go.Layout(scene = Scene, height = 1000,width = 1000)data = [trace]fig = go.Figure(data = data, layout = layout)fig.update_layout(title="'DBSCAN Clusters (6) Derived from PCA'", font=dict(size=12,))fig.show()
Before we get ahead of ourselves let’s quickly get a count of the number of employees in each cluster. Well it seems cluster 0 contains most of the data points which isn’t very informative. In fact, if we had ran the algorithm using epsilon values of 0.5 and minimum points of 5 which would have produced 63 clusters, cluster 0 still would have contained 99% of the employee population.
np.unique(labels, return_counts=True)
DBSCAN, a density clustering algorithm which is often used on non-linear or non-spherical datasets. Epsilon and Minimum Points are two required parameters. Epsilon is the radius within nearby data points that need to be in to be considered ‘similar’ enough to begin a cluster. Finally, Minimum Points is the minimum number of data points that need to be inside the radius (ie. epsilon) before they can be considered a cluster.
In our example, we attempted to cluster a dataset of 15,000 employees based on their job characteristics. We first standardized the dataset to scale the features. Next, we applied PCA in order to reduce the number of dimensions/features to 3 principal components. Using the “Elbow Method” we estimated an epsilon value of 0.2 and a minimum point value of 6. Using these parameters we were able to obtain 53 clusters, 1,500 outliers, and a silhouette score of -0.52. Needless to say, the results were not very promising.
Next, we attempted an iterative approach to fine-tuning the epsilon and minimum points parameters. We had decided upon an epsilon value of 1.0 and a minimum points value of 4. The algorithm returned 6 valid clusters (one -1 cluster), only 7 outliers, and a respectable silhouette score of 0.46. However, upon plotting the derived clusters it was discovered the first cluster contained 99% of the employees. Once again from a business perspective, we would want our clusters to be more equally distributed to provide us with good insights about our employees.
It would seem DBSCAN is not the optimal clustering algorithm for this particular dataset.
This same dataset was clustered using KMeans and produced much more equal and defined clusters (LINK).