K Means Clustering using PySpark on Big Data | by Angel Das | Towards Data Science

If you are not familiar with K Means clustering, I recommend going through the article below. This article focuses mostly on data parallelism and clustering, i.e. K Means clustering on Big Data.
towardsdatascience.com
Clustering is an unsupervised learning technique, in short, you are working on data, without having any information about a target attribute or a dependent variable. The general idea of clustering is to find some intrinsic structure in the data, often referred to as groups of similar objects. The algorithm studies the data to identify these patterns or groups such that each member in a group is closer to another member in the group (lower intracluster distance) and farther from another member in a different group (higher inter-cluster distance).
Most of you must be familiar with these examples from real life:
Customer Segmentation — Used widely for targeted marketingImage Segmentation — Identify the change in landscape or intrusionRecommendation Engines
Customer Segmentation — Used widely for targeted marketing
Image Segmentation — Identify the change in landscape or intrusion
Recommendation Engines
K Means clustering, irrespective of the platform uses a similarity measure in the form of Euclidean Distance. Often referred to as Divisive or Partitional Clustering, the basic idea of K Means is to start with every data point a bigger cluster and then divide them into smaller groups based on user input K (or the number of clusters). Each cluster has a center called the centroid. The total number of centroids in a given cluster is always equal to K. The algorithm iteratively looks for data points and assign them to their closest cluster.
Once all data points are assigned to their respective centroids (which represents each cluster here), the centroid values are recalculated and the process repeats until the clusters reach a convergence criterion. Centroids are nothing but new mean for each cluster (e.g. A Cluster Consisting of Customers A, B, C with an average spend of 100, 200, 300 and a basket size of 10, 15, and 20 will have centroids as 200 and 15 respectively). Convergence criterion is a measure of stability or compactness of the cluster, i.e. the intracluster distance between any two iterations doesn’t change beyond a given threshold.
Before we discuss why PySpark and not a Sklearn based algorithm, let’s discuss how the process differs in PySpark. When building any clustering algorithm using PySpark, one needs to perform a few data transformations. Let us dig deep into the data first. The data used for the analysis can be found here.
The dataset consists of 9K active credit cardholders over 6 months and their transaction and account attributes. The idea is to develop a customer segmentation for marketing strategy.
from pyspark.sql import SparkSessionspark = SparkSession.builder.appName(‘Clustering using K-Means’).getOrCreate()data_customer=spark.read.csv('CC General.csv', header=True, inferSchema=True)data_customer.printSchema()
The attributes that are self intuitive can be divided into three broader categories. Customer Information (Primary Key as CUST_ID), account information (balance, balance frequency, purchases, credit limit, tenure, etc.), and transactions (purchase frequency, payments, cash advance, etc.).
data_customer=data_customer.na.drop()
All attributes under consideration are numerical or discrete numeric, hence we need to convert them into features using a Vector Assembler. A vector assembler is a transformer that converts a set of features into a single vector column often referred to as an array of features. Features here are columns. Since customer id is an identifier that won’t be used for clustering, we first extract the required columns using .columns, pass it as an input to Vector Assembler, and then use the transform() to convert the input columns into a single vector column called a feature.
from pyspark.ml.feature import VectorAssemblerdata_customer.columnsassemble=VectorAssembler(inputCols=[ 'BALANCE', 'BALANCE_FREQUENCY', 'PURCHASES', 'ONEOFF_PURCHASES', 'INSTALLMENTS_PURCHASES', 'CASH_ADVANCE', 'PURCHASES_FREQUENCY', 'ONEOFF_PURCHASES_FREQUENCY', 'PURCHASES_INSTALLMENTS_FREQUENCY', 'CASH_ADVANCE_FREQUENCY', 'CASH_ADVANCE_TRX', 'PURCHASES_TRX', 'CREDIT_LIMIT', 'PAYMENTS', 'MINIMUM_PAYMENTS', 'PRC_FULL_PAYMENT', 'TENURE'], outputCol='features')assembled_data=assemble.transform(data_customer)assembled_data.show(2)
Now that all columns are transformed into a single feature vector we need to standardize the data to bring them to a comparable scale. E.g. Balance can have a scale from 10–1000 whereas balance frequency has a scale from 0–1 say. Euclidean distance is always impacted more by variables on a higher scale, hence it’s important to scale the variables out.
from pyspark.ml.feature import StandardScalerscale=StandardScaler(inputCol='features',outputCol='standardized')data_scale=scale.fit(assembled_data)data_scale_output=data_scale.transform(assembled_data)data_scale_output.show(2)
Now that our data is standardized we can develop the K Means algorithm.
K-means is one of the most commonly used clustering algorithms for grouping data into a predefined number of clusters. The spark.mllib includes a parallelized variant of the k-means++ method called kmeans||. The KMeans function from pyspark.ml.clustering includes the following parameters:
k is the number of clusters specified by the user
maxIterations is the maximum number of iterations before the clustering algorithm stops. Note that if the intracluster distance doesn’t change beyond the epsilon value mentioned, the iteration will stop irrespective of max iterations
initializationMode specifies either random initialization of centroids or initialization via k-means|| (similar to K-means ++)
epsilon determines the distance threshold within which k-means is expected to converge
initialModel is an optional set of cluster centroids that the user can provide as an input. If this parameter is used, the algorithm just runs once to allocate points to its nearest centroid
train(k=4, maxIterations=20, minDivisibleClusterSize=1.0, seed=-1888008604) are the default values.
from pyspark.ml.clustering import KMeansfrom pyspark.ml.evaluation import ClusteringEvaluatorsilhouette_score=[]evaluator = ClusteringEvaluator(predictionCol='prediction', featuresCol='standardized', \                                metricName='silhouette', distanceMeasure='squaredEuclidean')for i in range(2,10):        KMeans_algo=KMeans(featuresCol='standardized', k=i)        KMeans_fit=KMeans_algo.fit(data_scale_output)        output=KMeans_fit.transform(data_scale_output)                score=evaluator.evaluate(output)        silhouette_score.append(score)        print("Silhouette Score:",score)
Visualizing the silhouette score. Note previous versions of the K Means had computeScore() that calculated the sum of intracluster distance but got deprecated in Spark 3.0.0. Silhouette Score using ClusteringEvaluator() measures how close each point in one cluster is to points in the neighboring clusters thus helping in figuring out clusters that are compact and well-spaced out.
#Visualizing the silhouette scores in a plotimport matplotlib.pyplot as pltfig, ax = plt.subplots(1,1, figsize =(8,6))ax.plot(range(2,10),silhouette_score)ax.set_xlabel(‘k’)ax.set_ylabel(‘cost’)
I prefer going ahead with K=7 where a local maxima of Silhouette Score is observed. Since there is no correct answer for what value of K is good, we can restore to descriptive statistics and plots to check the distribution of customers. This is where SkLearn comes in more handy with documentation and PCA implementation. Most of us are more comfortable studying an elbow plot with SSE on the y axis rather than a Silhouette Score, but PySpark comes with its merit.
PySpark uses the concept of Data Parallelism or Result Parallelism when performing the K Means clustering. Imagine you need to roll out targeted marketing campaigns for the Boxing Day event in Melbourne and you want to reach out to 200K customers with different purchasing attributes. Imagine running multiple iterations of K Means on your local system. For K=5, the number of distance metrics you require to calculate is 5 x 200K = 1Million. 1 Million such metrics need to be calculated for say 30 times before a convergence criterion is met, i.e. 30Million distances (euclidean). You will require a lot of computation power and time to handle a scenario like this.
What data parallelism does is, creates parallelism from the beginning by partitioning the dataset into smaller partitions. On the other hand, result parallelism is based on targeted clusters. Let’s say:
D = Number of Records {X1, X2, ......, Xn}
k = Number of clusters
P = Number of Processors {P1, P2,...Pm}
C = Initial Centroids {C1, C2, .... Ck}
Data D is divided across P processors. Each processor works on a set of records (determined by spark configuration). The initial centroid values, C, are shared across each of these processorsNow every processor has centroid information. The processors compute the distance of their records to these centroids and form local clusters by assigning data points to its closest centroidOnce step 2 is done, the sum and count of records for each of these clusters across P processors are stored by a master process for future referenceOnce one iteration is completed, the information from the processor is exchanged and a master process computes the updated Centroids and shares them across the P processors again, i.e. one point is allocated to K clusters, a master process updates the centroids, and reshare the information back with the processorsThis process keeps iterating till convergence is met. Once a convergence criterion is met, the master process collects local clusters and combines them into a global one
Data D is divided across P processors. Each processor works on a set of records (determined by spark configuration). The initial centroid values, C, are shared across each of these processors
Now every processor has centroid information. The processors compute the distance of their records to these centroids and form local clusters by assigning data points to its closest centroid
Once step 2 is done, the sum and count of records for each of these clusters across P processors are stored by a master process for future reference
Once one iteration is completed, the information from the processor is exchanged and a master process computes the updated Centroids and shares them across the P processors again, i.e. one point is allocated to K clusters, a master process updates the centroids, and reshare the information back with the processors
This process keeps iterating till convergence is met. Once a convergence criterion is met, the master process collects local clusters and combines them into a global one
Imagine 200K records divided across 3 processors each having ~70K records. This is where distributed processing comes into the picture to reduce the volume of data, yet ensuring a complete result.
In result parallelism say:
D = Number of Records {X1, X2, ......, Xn}
k = Number of clusters
P = Number of Processors {P1, P2,...Pm}
C = Initial Centroids {C1, C2, .... Ck}
Data D is divided across P processors and then sorted within each processor. Each processor works on a set of records (determined by spark configuration)The initial centroid values, C, are initialized and divided/shared across each of these processors (i.e., unlike data parallelism where all centroid values are shared across all processors, here, we pass one centroid value to one processorNow every processor has one centroid of information. Compute the distance of these points to these centroids. For data points in a processor which are extremely low or high: If they are closer to the centroid of processor assign them to that cluster, else if they are closer to the centroid belonging to a different processor, move data point to the new processorRepeat till convergence is met. All local clusters from Processor P is returned
Data D is divided across P processors and then sorted within each processor. Each processor works on a set of records (determined by spark configuration)
The initial centroid values, C, are initialized and divided/shared across each of these processors (i.e., unlike data parallelism where all centroid values are shared across all processors, here, we pass one centroid value to one processor
Now every processor has one centroid of information. Compute the distance of these points to these centroids. For data points in a processor which are extremely low or high: If they are closer to the centroid of processor assign them to that cluster, else if they are closer to the centroid belonging to a different processor, move data point to the new processor
Repeat till convergence is met. All local clusters from Processor P is returned
https://spark.apache.org/docs/latest/mllib-clustering.htmlhttps://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.clustering.KMeansModelhttps://spark.apache.org/docs/latest/api/python/pyspark.ml.html#pyspark.ml.evaluation.ClusteringEvaluatorhttps://spark.apache.org/docs/latest/api/python/_modules/pyspark/ml/evaluation.html
https://spark.apache.org/docs/latest/mllib-clustering.html
https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.clustering.KMeansModel
https://spark.apache.org/docs/latest/api/python/pyspark.ml.html#pyspark.ml.evaluation.ClusteringEvaluator
https://spark.apache.org/docs/latest/api/python/_modules/pyspark/ml/evaluation.html
About the Author: Advanced analytics professional and management consultant helping companies find solutions for diverse problems through a mix of business, technology, and math on organizational data. A Data Science enthusiast, here to share, learn and contribute; You can connect with me on Linked and Twitter;