An Implementation of DBSCAN on PySpark | by Salil Jain | Towards Data Science

DBSCAN is a well-known clustering algorithm that has stood the test of time. Though the algorithm is not included in Spark MLLib. There are a few implementations (1, 2, 3) though they are in scala. Implementation in PySpark uses the cartesian product of rdd to itself which results in O(n2) complexity and possibly O(n2) memory before the filter.
ptsFullNeighborRDD=rdd.cartesian(rdd)                            .filter(lambda (pt1,pt2): dist(pt1,pt2)<eps)                            .map(lambda (pt1,pt2):(pt1,[pt2]))                            .reduceByKey(lambda pts1,pts2: pts1+pts2)                            .filter(lambda (pt, pts): len(pts)>=minPts)source: https://github.com/htleeab/DBSCAN-pyspark/blob/master/DBSCAN.py
For a quick primer on the complexity of the DBSCAN algorithm:
https://en.wikipedia.org/wiki/DBSCAN#Complexity
DBSCAN visits each point of the database, possibly multiple times (e.g., as candidates to different clusters). For practical considerations, however, the time complexity is mostly governed by the number of regionQuery invocations. DBSCAN executes exactly one such query for each point, and if an indexing structure is used that executes a neighborhood query in O(log n), an overall average runtime complexity of O(n log n) is obtained (if parameter ε is chosen in a meaningful way, i.e. such that on average only O(log n) points are returned). Without the use of an accelerating index structure, or on degenerated data (e.g. all points within a distance less than ε), the worst case run time complexity remains O(n2). The distance matrix of size (n2-n)/2 can be materialized to avoid distance recomputations, but this needs O(n2) memory, whereas a non-matrix based implementation of DBSCAN only needs O(n) memory.
In this post, we will explore how we can implement DBSCAN in PySpark efficiently without using O(n2) operations by reducing the number of distance calculations. We would implement an indexing/partitioning structure based on Triangle Inequality to achieve this.
Let’s refresh triangle inequality. If there are three vertices of the triangle a, b and c, and given distance metric d. Then
d(a, b) ≤ d(a, c) + d(c, b)
d(a, c) ≤ d(a, b) + d(b, c)
d(b, c) ≤ d(b, a) + d(a, c)
In DBSCAN there is a parameter ε, which is used to find the linkage between points. Now, let us use this parameter to see if we can use triangle inequality to reduce the number of operations.
Let’s say there are four points x, y, z, and c.
Lemma 1: If d(x, c) ≥ (k+1)ε and d(y, c) < kε then d(x, y) > ε
As per triangle inequality,
d(x, c) ≤ d(x, y) + d(y, c)
d(x, c)-d(y, c)≤ d(x, y)
d(x, c)-d(y, c) > (k+1)ε -kε > ε
so d(x, y) > ε
Lemma 2: If d(x, c) ≤ lε and d(z, c) > (l+1)ε then d(x, z) > ε
As per triangle inequality,
d(z, c) ≤ d(x, z) + d(x, c)
d(z, c)-d(x, c)≤ d(x, z)
d(z, c)-d(x, c) > (l+1)ε -lε
d(z, c)-d(x, c) > ε
so d(x, z) > ε
What we can deduce from above is that if we compute distances of all points from c then we can filter points y and z using above criteria. We can compute distances from c and partition points in concentric rings (center being c).
From the above lemmas, we can see that if
(m+1)ε ≥ d(x, c) ≥ mε then we can filter out points y and z if d(y, c) < (m-1)ε and d(z, c) > (m+2)ε
From this, we can deduce that for any point for which (m+1)ε ≥ d(x, c) ≥ mε is true we can have a partition of (m+3)ε width starting at (m-1)ε and ending at (m+2)ε.
Here is a depiction how it would look like. Two-dimensional space is divided into quantiles of ε euclidean distance. Green rings show the partition. x1 is at the (m+1/2)ε distance from c (center of (m-1)ε — (m+2)ε partition). x2 and x3 are at mε and (m+1)ε distance from c. It is clear that for x1, x2, and x3 all points of relevance (within the circle of radius ε) are within the partition.
If we create mutually exclusive partitions and compute distances among points within that partition, it would be incomplete. For example, x4 and x5’s range circle will overlap two partitions. Hence the need for overlapping partition. One strategy could be to move the partition by ε. Though in that case if partition width is 3ε, then one point may occur in three different partitions. Instead, partitions are created of 2ε width and move them by ε. In this case, one point may occur only in two partitions.
The above two images show how this partition scheme works. Two partitions in combination allow a range query of ε radius for all the points from mε to (m+1)ε. The first partition covers all the points from mε to (m+1/2)ε (x2 covered but x3 not) and the second covers (m+1/2)ε to (m+1)ε (x3 covered but x2 not).
Let us see how these partitions look like on some generated data.
The above data and image are generated by the following code:
The above partitions from data are generated by the following code:
partition_index identifies each partition. Each data point is put into two partitions as discussed before based on the distance from c (pivot) and ε. distance method handles one point at a time. In PySpark flatMap method is used to map every point into an array of tuples (out).
All data points within a partition are merged before generating the visualization. They also need to be merged for further processing DBSCAN on PySpark.
reduceByKey method is used to merge partition data in one. The use of word partition may confuse with PySpark partitions but those are two separate things. Though partitionBy method could be used to reconcile that as well.
After reduceByKey, we will get each row of the rdd as a ring depicted in figure 4. As you can see there is an overlap so points would be in two rows of rdd and that is intentional.
The above code computes distance within each partition. The output of the method is a list of tuples. Each tuple has the id of point and set of its neighbors within ε distance. As we know that point would occur in two partitions so we need to combine the sets for a given point so we get all of its neighbors within ε distance in whole data. reduceByKey is used to combine the sets by doing union operation on them.
reduceByKey(lambda x, y: x.union(y))
Combined code till now looks as follows:
Once we have neighbors within ε distance for a point we can identify if it is a core or border point.
Core Point: There are at least min_pts within ε distance
Border Point: There are less than min_pts within ε distance but one of them is the core point.
To identify core and border points, first, we assign them to a cluster. For each point which is a core point, we create a cluster label the same to its id (assuming the ids are unique). We create a tuple for each core point and its neighbors of the form (id, [(cluster_label, is_core_point)]). All the neighbors in this scenario would be labeled as a base point. Let's take an example
min_pts = 3Input: (3, set(4,6,2))Output: [(3, [(3, True)]), (4, [(3, False)]), (6, [(3, False)]), (2, [(3, False)])]
Input is a tuple where 3 is the id of point and (4, 6, 2) are its neighbors within ε distance.
As one can see all points are assigned cluster label 3. While 3 is assigned True for is_core_point as a core point and all other points are considered base points and assigned False for is_core_point.
We may have similar input tuples for 4, 6, and 2 where they may or may not be assigned as core points. The idea is to eventually combine all cluster labels for a point and if at least one of the assignment for is_core_point is True then its a core point otherwise its a border point.
We combine all (cluster_label, is_core_point) tuples for a point using reduceByKey method and then investigate if its a core point or not while combining all clusters labels for that point. If its a border point then we would only leave one cluster label for it.
The above method is used to combine all cluster label for a point. Again if it is a border point then we return only 1st cluster label.
Code in PySpark till now looks like the following:
Now for each point, we have cluster labels. If a point has more than one cluster label then it means those clusters are connected. Those connected clusters are the final clusters we need to solve DBSCAN. We solve this by creating a graph with vertices as cluster labels and edges between cluster labels if they are assigned to the same point.
In the above code, combine_cluster_rdd is a collection of rows where each row is a tuple (point, cluster_labels). Each cluster label is vertices and combinations of cluster labels for a point are the edges. Connected components of this graph give a mapping between each cluster label and a connected cluster. which we can apply to points to get final clusters.
Above is how the final method looks like which returns a Spark Dataframe with point id, cluster component label, and a boolean indicator if its a core point.
Now I compare the results in terms of accuracy with sklearn implementation of DBSCAN.
The make_blobs method creates blobs around three input centers. Running DBSCAN using sklearn and my implementation with ε=0.3 and min_pts=10 gives the following results.
Core points are bigger circles while border points are smaller ones. Noise points are colored black which is the same in both implementations. One thing that jumps out is the border points are assigned different clusters that speak to the non-deterministic nature of DBSCAN. My other post also talks about it.
medium.com
For ε=0.2 we get the border points assigned to the same clusters. Following is some code and results on data in rings.
For n=750, the number of distance operations required by a simple implementation of DBSCAN would be n(n-1)/2 which is 280875. As we create the partition based on ε, the smaller the ε less the number of distance operations would be needed. In this case, there were a total of 149716(ε=.2) and 217624(ε=0.3) operations needed.
A pyspark implementation which would be efficient based on the value of ε with the following steps:
Partitioning Data: Partition with overlapping rings of 2ε width moved by ε.Merging Partition Data: So we get all partition data in a single record.Distance Calculations: Calculate distance within the same partitionPoint Labeling: Based on the number of neighbors, label core, and border points.Connected Clusters: Use Graphframe to connect clusters labels to evaluate final DBSCAN labels.
Partitioning Data: Partition with overlapping rings of 2ε width moved by ε.
Merging Partition Data: So we get all partition data in a single record.
Distance Calculations: Calculate distance within the same partition
Point Labeling: Based on the number of neighbors, label core, and border points.
Connected Clusters: Use Graphframe to connect clusters labels to evaluate final DBSCAN labels.
A comparison with the existing implementation shows the accuracy of the algorithm and the implementation of this post.
On the local machine with both driver and worker nodes, implementation is slower than sklearn. There could be a couple of reasons which need to be investigated:
For a small amount of data, sklearn may be much faster but is it the case for big-data?Graphframe takes quite some time to execute and wondering if connected components analysis can be performed on the driver with some other graph library?
For a small amount of data, sklearn may be much faster but is it the case for big-data?
Graphframe takes quite some time to execute and wondering if connected components analysis can be performed on the driver with some other graph library?
Complete PySpark Implementation can be fount at: