How to Use Pyspark For Your Machine Learning Project | by François St-Amant | Towards Data Science

Pyspark is a Python API that supports Apache Spark, a distributed framework made for handling big data analysis. It’s an amazing framework to use when you are working with huge datasets, and it’s becoming a must-have skill for any data scientist.
In this tutorial, I will present how to use Pyspark to do exactly what you are used to see in a Kaggle notebook (cleaning, EDA, feature engineering and building models).
I used a database containing information about customers for a telecom company. The objective is to predict which clients will leave (Churn) in the upcoming three months. The CSV file with the data contains more than 800,000 rows and 8 features, as well as a binary Churn variable.
The goal here is not to find the best solution. It’s rather to show you how to work with Pyspark. Along the way I will try to present many functions that can be used for all stages of your machine learning project!
Let’s begin by creating a SparkSession, which is the entry point to any Spark functionality.
import pysparkfrom pyspark.sql import SparkSessionspark = SparkSession.builder.master("local[4]")\       .appName("test").getOrCreate()
Here is how to read a CSV using Pyspark.
df=spark.read.csv('train.csv',header=True,sep= ",",inferSchema=True)
Here is what the dataframe looks like.
The Pyspark.sql module allows you to do in Pyspark pretty much anything that can be done with SQL.
For instance, let’s begin by cleaning the data a bit. First, as you can see in the image above, we have some Null values. I will drop all rows that contain a null value.
df = df.na.drop()
Then, thewhen/otherwise functions allow you to filter a column and assign a new value based on what is found in each row. Here for instance, I replace Male and Female with 0 and 1 for the Sex variable.
from pyspark.sql.functions import whendf = df.withColumn("gender",when(df["gender"]=='M',0).otherwise(1))
Once the data is all cleaned up, many SQL-like functions can help analyze it.
For instance, the groupBy function allows you to group values and return count, sum or whatever for each category. Let’s see how many data points belong to each class for the churn variable.
df.groupBy('churnIn3Month').count().show()
We have imbalanced classes here. While I will not do anything about it in this tutorial, in an upcoming one, I will show you how to deal with imbalanced classes using Pyspark, doing things like undersampling, oversampling and SMOTE.
Another interesting thing to do is to look at how certain features vary between the two groups (clients that left and the ones that did not). Here is one interesting result I found.
from pyspark.sql.functions import avgdf.select("phoneBalance","churnIn3Month").\          groupBy("ChurnIn3Month").agg(avg("phoneBalance"))
We see that customers that left had on average a much smaller phone balance, which means their phone was much closer to being paid entirely (which makes it easier to leave a phone company of course).
Now, let’s look at a correlation matrix. I created it using the correlation function in Pyspark. I also cheated a bit and used Pandas here, just to easily create something much more visual.
from pyspark.ml.stat import Correlationx=df.columns[2:11]corr_plot = pd.DataFrame()for i in x:    corr=[]    for j in x:        corr.append(round(df.stat.corr(i,j),2))    corr_plot = pd.concat([corr_plot,pd.Series(corr)],axis=1)corr_plot.columns=xcorr_plot.insert(0,'',x)corr_plot.set_index('')
Again, phoneBalance has the strongest correlation with the churn variable.
Considering the results from above, I decided to create a new variable, which will be the square of thephoneBalance variable. Here is how to do that with Pyspark.
from pyspark.sql.functions import col, powdf = df.withColumn('phoneBalance2',pow(col('phoneBalance'),2))
The withColumn function allows you to add columns to your pyspark dataframe. Super useful!
The Machine Learning library in Pyspark certainly is not yet to the standard of Scikit Learn. With that being said, you can still do a lot of stuff with it.
The first thing you have to do however is to create a vector containing all your features. All the methods we will use require it.
from pyspark.ml.feature import VectorAssemblerignore=['churnIn3Month', 'ID','_c0']vectorAssembler = VectorAssembler(inputCols=[x for x in df.columns                    if x not in ignore], outputCol = 'features')new_df = vectorAssembler.transform(df)new_df = new_df.select(['features', 'churnIn3Month'])
Then, let’s split the data into a training and validation set.
train, test = new_df.randomSplit([0.75, 0.25], seed = 12345)
Alright, now let’s build some models. I will only show a couple models, just to give you an idea of how to do it with Pyspark.
from pyspark.ml.classification import LogisticRegressionlr = LogisticRegression(featuresCol = 'features',                          labelCol='churnIn3Month')lr_model = lr.fit(train)
We can look at the ROC curve for the model. As a reminder, the closer the AUC (area under the curve) is to 1, the better the model is at distinguishing between classes.
import matplotlib.pyplot as pltplt.plot(lr_model.summary.roc.select('FPR').collect(),         lr_model.summary.roc.select('TPR').collect())plt.xlabel('False Positive Rate')plt.ylabel('True Positive Rate')plt.show()lr_model.summary.areaUnderROC
Let’s do one more model, to showcase how easy it can be to fit models once the data is put in the right format for Pyspark, i.e. vectors. Here is how to create a random forest model.
from pyspark.ml.classification import RandomForestClassifierrf = RandomForestClassifier(featuresCol = 'features', labelCol =                             'churnIn3Month')rf_model = rf.fit(train)
And here is how to get the AUC for the model:
from pyspark.ml.evaluation import BinaryClassificationEvaluatorpredictions = rf_model.transform(test)auc = BinaryClassificationEvaluator().setLabelCol('churnIn3Month')print('AUC of the model:' + str(auc.evaluate(predictions)))
Both models are very similiar, but the results suggest that the logistic regression model is slightly better in our case.
Scikit Learn is fantastic and will perform admirably, for as long as you are not working with too much data. Sadly, the bigger your projects, the more likely it is that you will need Spark. Thankfully, as you have seen here, the learning curve to start using Pyspark really isn’t that steep, especially if you are familiar with Python and SQL. In my mind, the main weakness of Pyspark is data visualization, but hopefully with time that will change!
There you have it. This article should serve as a great starting point for anyone that wants to do Machine Learning with Pyspark. I hope you liked it and thanks for reading!