Building an ML application using MLlib in Pyspark | by Yashwanth Madaka | Towards Data Science

Apache Spark is one of the on-demand big data tools which is being used by many companies around the world. Its ability to do In-Memory computation and Parallel-Processing are the main reasons for the popularity of this tool.
MLlib is a scalable Machine learning library which is present alongside other services like Spark SQL, Spark Streaming and GraphX on top of Spark.
In this article, we are going to concentrate on a dataset called Stroke dataset. Stoke is a condition in which either the blood flow to the brain stops or blood flow is excessive. The risk factors of stroke are
cigarette smoking
high blood pressure
diabetes
high blood cholesterol levels
heavy drinking
a diet high in fat (particularly saturated) and salt, but low in fiber, fruit, and vegetables
lack of regular exercise
obesity
So I got a good dataset here: https://bigml.com/user/francisco/gallery/model/508b2008570c6372100000b1#info
The below is the dataset:
The dataset contains almost all the risk factors of the stroke mentioned above. Hence it is important to choose a dataset with appropriate risk factors.
We are going to change the string values of the columns into a numerical values. The reason for this will be explained later. Using replace function in Excel, I had changed the dataset into the below
Gender column — Male=1, Female=0
Gender column — Male=1, Female=0
2. Smoking history — Never=0, Ever=0.25, Not current = 0.5, former = 0.75, current = 1.0
Google cloud — we will be setting up our spark clusters in Dataproc and code in Jupyter notebookJpmml(pyspark2pmml) — which is used to convert our model into and pmml file.Openscoring — A REST web service for scoring PMML models.VS Code — We will be using React JS to build an interactive website which communicates with the REST server.
Google cloud — we will be setting up our spark clusters in Dataproc and code in Jupyter notebook
Jpmml(pyspark2pmml) — which is used to convert our model into and pmml file.
Openscoring — A REST web service for scoring PMML models.
VS Code — We will be using React JS to build an interactive website which communicates with the REST server.
The below image showcases a brief architecture of our whole project.
Google cloud has a service called Dataproc which is used to create clusters which come preinstalled with Apache Spark. We can resize our clusters anytime we want to. Google cloud provides free $300 credit as an introductory offer. So, we will be using these free credits to set up our cluster.
Click on ‘Activate’ to get the free $300 credit.
Select your country and click on “continue”. In the next page, you will be prompted to enter your billing details and credit or debit card details. Fill them and click the bottom button.
Console page will be opened. On top of the page, type Dataproc in the search bar and the above page will be opened. Click on create a cluster to start creating a cluster.
Make sure you enter the same above settings. Click on Advanced options and follow the above image settings and click on create. It might take 2 to 3 mins to create a cluster.
Navigate to the cluster and click on VM instances. Under the VM instances, we can see that a master node and two worker nodes are created. The role of the master node is it usually requests the resources in the cluster and makes them available to the spark driver. It monitors and tracks the status of worker nodes whose work is to host the executor process which stores output data from tasks and also hosts the JVM. A detailed description can be found here
Now click on the SSH button of the master node.
A new terminal is opened in a new chrome tab. This is the command line interface through which we can interact with our cluster. Type “pyspark” to check the installation on spark and its version. Make sure the version of spark is above 2.2 and python version is 3.6.
Now to setup jupyter notebook, we need to create a firewall rule. Follow the images to setup new firewall rule. Make sure you select “Allow all” in the protocols and ports.
Click on save and navigate to “External IP addresses”.
change the TYPE of ‘spark-cluster-m’ to static. Give any name and click on “RESERVE”.
Now navigate to “SSH” and type the below commands.
sudo nano ~/.jupyter_notebook_config.py
copy the below lines and paste it. Press CTRL+o, Enter, CTRL+x.
c=get_config()c.NotebookApp.ip=’*’c.NotebookApp.open_browser=Falsec.NotebookApp.port=5000
Now we can open jupyter notebook by below command
jupyter-notebook --no-browser — port=5000
type the above command in the SSH and then open a new tab and type “https://localhost:5000” in your google chrome to open Jupyter notebook. In my case, the localhost is 35.230.35.117
Before going into this section, we need to install a few external libraries.
We need Imblearn library to perform SMOTE as our dataset is highly imbalanced. More information about smote can be found in this link.
In SSH, type
sudo -i
and then type the below line
conda install -c glemaitre imbalanced-learn
exit from the root folder and then open Jupyter notebook. Let’s start coding.
Import important libraries
from pyspark.sql import SQLContextfrom pyspark.sql import DataFrameNaFunctionsfrom pyspark.ml import Pipelinefrom pyspark.ml.classification import DecisionTreeClassifierfrom pyspark.ml.classification import LogisticRegressionfrom pyspark.ml.feature import Binarizerfrom pyspark.ml.feature import OneHotEncoder, VectorAssembler, StringIndexer, VectorIndexerfrom pyspark.ml.classification import RandomForestClassifierfrom pyspark.sql.functions import avgimport pandas as pdimport numpy as npimport matplotlib.pyplot as plt%matplotlib inlinefrom pyspark.ml.evaluation import MulticlassClassificationEvaluatorfrom pyspark.mllib.evaluation import MulticlassMetricsfrom pyspark.ml.evaluation import BinaryClassificationEvaluatorfrom imblearn.over_sampling import SMOTEfrom imblearn.combine import SMOTEENNfrom sklearn.model_selection import train_test_splitfrom collections import Counter
Now we need to create a spark session.
from pyspark.context import SparkContextfrom pyspark.sql.session import SparkSessionsc = SparkContext(‘local’)spark = SparkSession(sc)
We need to access our datafile from storage. Navigate to “bucket” in google cloud console and create a new bucket. I had given the name “data-stroke-1” and upload the modified CSV file.
Now we need to load the CSV file which we had uploaded in our bucket.
input_dir = ‘gs://data-stroke-1/’df = spark.read.format(‘com.databricks.spark.csv’).options(header=’true’, inferschema=’true’).load(input_dir+’stroke.csv’)df.columns
We can check our dataframe by printing it using the command shown in the below figure.
Now, we need to create a column in which we have all the features responsible to predict the occurrence of stroke.
featureColumns = [‘gender’,’age’,‘diabetes’,‘hypertension’, ‘heart disease’,‘smoking history’,‘BMI’]
Make sure all the columns are in double values. Next, let's remove all the entries which are under age below 2.
df = df.filter(df.age >2)df.count()
Now let's print a bar graph to check the type of classes present in our data
responses = df.groupBy(‘stroke’).count().collect()categories = [i[0] for i in responses]counts = [i[1] for i in responses] ind = np.array(range(len(categories)))width = 0.35plt.bar(ind, counts, width=width, color=’r’) plt.ylabel(‘counts’)plt.title(‘Stroke’)plt.xticks(ind + width/2., categories)
Step3A. Missing data Management
Now, it's very important to do proper missing data management to get a very good model at the end. It's not always good to use “df.na.drop()” which removes all the rows which have missing data. Filling them with proper reasonable values is one idea which we can implement.
As we can see, we have missing values in BMI column and Smoking history column. One possible way to fill up these BMI values is to use the Age values to fill them up.
For smoking history, it is very difficult to find reasonable values to fill them up. Usually, people below 16 years are not that addicted to smoking and hence we can fill up the values of those age group people with 0. People between age 17 to 24 might have tried smoking at least once in their life and so we can give 0.25 value to those people. Now a few people after a certain age quit and few of them still continue even though they have health problems. We cannot decide which value to give to them and so by default, we will give assign the value 0 to them.
We can either drop all the rows which have missing values in these columns or we can fill in those by the above logic. But for the purpose of this tutorial, I had filled the missing rows by the above logic but practically tampering with the data with no data-driven logic to back it up is usually not a good idea.
We are going to perform a few operations on this dataframe and spark dataframe doesn’t support any operations. So we replicate our dataframe to pandas dataframe and then perform the actions.
imputeDF = dfimputeDF_Pandas = imputeDF.toPandas()
We will be dividing the full dataframe into many dataframes based on the age and fill them with reasonable values and then, later on, combine all the dataframes into one and convert it back to spark dataframe.
df_2_9 = imputeDF_Pandas[(imputeDF_Pandas[‘age’] >=2 ) & (imputeDF_Pandas[‘age’] <= 9)]values = {‘smoking history’: 0, ‘BMI’:17.125}df_2_9 = df_2_9.fillna(value = values)df_10_13 = imputeDF_Pandas[(imputeDF_Pandas[‘age’] >=10 ) & (imputeDF_Pandas[‘age’] <= 13)]values = {‘smoking history’: 0, ‘BMI’:19.5}df_10_13 = df_10_13.fillna(value = values)df_14_17 = imputeDF_Pandas[(imputeDF_Pandas[‘age’] >=14 ) & (imputeDF_Pandas[‘age’] <= 17)]values = {‘smoking history’: 0, ‘BMI’:23.05}df_14_17 = df_14_17.fillna(value = values)df_18_24 = imputeDF_Pandas[(imputeDF_Pandas[‘age’] >=18 ) & (imputeDF_Pandas[‘age’] <= 24)]values = {‘smoking history’: 0, ‘BMI’:27.1}df_18_24 = df_18_24.fillna(value = values)df_25_29 = imputeDF_Pandas[(imputeDF_Pandas[‘age’] >=25 ) & (imputeDF_Pandas[‘age’] <= 29)]values = {‘smoking history’: 0, ‘BMI’:27.9}df_25_29 = df_25_29.fillna(value = values)df_30_34 = imputeDF_Pandas[(imputeDF_Pandas[‘age’] >=30 ) & (imputeDF_Pandas[‘age’] <= 34)]values = {‘smoking history’: 0.25, ‘BMI’:29.6}df_30_34 = df_30_34.fillna(value = values)df_35_44 = imputeDF_Pandas[(imputeDF_Pandas[‘age’] >=35 ) & (imputeDF_Pandas[‘age’] <= 44)]values = {‘smoking history’: 0.25, ‘BMI’:30.15}df_35_44 = df_35_44.fillna(value = values)df_45_49 = imputeDF_Pandas[(imputeDF_Pandas[‘age’] >=45 ) & (imputeDF_Pandas[‘age’] <= 49)]values = {‘smoking history’: 0, ‘BMI’:29.7}df_45_49 = df_45_49.fillna(value = values)df_50_59 = imputeDF_Pandas[(imputeDF_Pandas[‘age’] >=50 ) & (imputeDF_Pandas[‘age’] <= 59)]values = {‘smoking history’: 0, ‘BMI’:29.95}df_50_59 = df_50_59.fillna(value = values)df_60_74 = imputeDF_Pandas[(imputeDF_Pandas[‘age’] >=60 ) & (imputeDF_Pandas[‘age’] <= 74)]values = {‘smoking history’: 0, ‘BMI’:30.1}df_60_74 = df_60_74.fillna(value = values)df_75_plus = imputeDF_Pandas[(imputeDF_Pandas[‘age’] >75 )]values = {‘smoking history’: 0, ‘BMI’:28.1}df_75_plus = df_75_plus.fillna(value = values)
Combine all dataframes
all_frames = [df_2_9, df_10_13, df_14_17, df_18_24, df_25_29, df_30_34, df_35_44, df_45_49, df_50_59, df_60_74, df_75_plus]df_combined = pd.concat(all_frames)df_combined_converted = spark.createDataFrame(df_combined)imputeDF = df_combined_converted
Step 3B. Dealing with imbalanced data
We will perform SMOTE technique to deal with imbalanced data. SMOTE can be referred from here:
X = imputeDF.toPandas().filter(items=[‘gender’, ‘age’, ‘diabetes’,’hypertension’,’heart disease’,’smoking history’,’BMI’])Y = imputeDF.toPandas().filter(items=[‘stroke’])X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.1, random_state=0)
sm = SMOTE(random_state=12, ratio = ‘auto’, kind = ‘regular’)x_train_res, y_train_res = sm.fit_sample(X_train, Y_train)print(‘Resampled dataset shape {}’.format(Counter(y_train_res)))
Please refer to this link to know about the parameters
X_train contains all the columns of the data except the Stroke column.
Y_train contains the Stroke column data.
Combine the resampled data into one spark dataframe
dataframe_1 = pd.DataFrame(x_train_res,columns=[‘gender’, ‘age’, ‘diabetes’, ‘hypertension’, ‘heart disease’, ‘smoking history’, ‘BMI’])dataframe_2 = pd.DataFrame(y_train_res, columns = [‘stroke’])# frames = [dataframe_1, dataframe_2]result = dataframe_1.combine_first(dataframe_2)
change it back to spark dataframe
imputeDF_1 = spark.createDataFrame(result)
Check the Resampled data. Its the same code we used earlier.
As we can see, we resampled our data successfully. Now we will be getting into the next part.
Below is a common pipeline of a spark ml project except in our case we had not used string indexer and oneHotEncoder.
Now to build an assembler and for this, we need a Binarizer.
binarizer = Binarizer(threshold=0.0, inputCol=”stroke”, outputCol=”label”)binarizedDF = binarizer.transform(imputeDF_1)binarizedDF = binarizedDF.drop(‘stroke’)
What this does is it creates a new column with name “label” and values the same as the values in stroke column.
assembler = VectorAssembler(inputCols = featureColumns, outputCol = “features”)assembled = assembler.transform(binarizedDF)print(assembled)
Assembler assembles all the columns which are required to predict stroke and produces one vector called features.
And now over to Splitting Data
(trainingData, testData) = assembled.randomSplit([0.7, 0.3], seed=0)print(“Distribution of Ones and Zeros in trainingData is: “, trainingData.groupBy(“label”).count().take(3))
Training
dt = DecisionTreeClassifier(labelCol="label", featuresCol="features", maxDepth=25, minInstancesPerNode=30, impurity="gini")pipeline = Pipeline(stages=[dt])model = pipeline.fit(trainingData)
Testing
predictions = model.transform(testData)
AUC-ROC
from pyspark.mllib.evaluation import BinaryClassificationMetrics as metricresults = predictions.select(['probability', 'label']) ## prepare score-label setresults_collect = results.collect()results_list = [(float(i[0][0]), 1.0-float(i[1])) for i in results_collect]scoreAndLabels = sc.parallelize(results_list) metrics = metric(scoreAndLabels)print("Test Data Aread under ROC score is : ", metrics.areaUnderROC)
from sklearn.metrics import roc_curve, auc fpr = dict()tpr = dict()roc_auc = dict() y_test = [i[1] for i in results_list]y_score = [i[0] for i in results_list] fpr, tpr, _ = roc_curve(y_test, y_score)roc_auc = auc(fpr, tpr) %matplotlib inlineplt.figure()plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % roc_auc)plt.plot([0, 1], [0, 1], 'k--')plt.xlim([0.0, 1.0])plt.ylim([0.0, 1.05])plt.xlabel('False Positive Rate')plt.ylabel('True Positive Rate')plt.title('Receiver operating characteristic Graph')plt.legend(loc="lower right")plt.show()
As we can see, we got the score of AUC-ROC around 98 which is very good. The model might be overfitted due to the usage of SMOTE technique. (But Logistic Regression works well for this dataset. But there seem to be some bugs in pyspark2pmml library which doesn’t export the logistic regression model properly.) So for demonstration purpose I am going to use Decision Tree model file.
To do this, we are going to use a library called PySPark2PMML and its details can be found here (https://github.com/jpmml/pyspark2pmml)
Save the jupyter file and exit the jupyter notebook.
Download the jpmml-sparkml-executable-1.5.3.jar file from — https://github.com/jpmml/jpmml-sparkml/releases
Upload it into the SSH
After upload, if we check by running “ls” command we see our file.
Now we need to setup jupyter notebook in such a way that when we type pyspark in the ssh, we need to fire open jupyter notebook. To do so, we need to change the environment variable.
Update PySpark driver environment variables:
add the below lines to your ~/.bashrc (or ~/.zshrc) file. Press “i” to insert the new lines. Copy the below code and paste it using “CTRL+V”.
export PYSPARK_DRIVER_PYTHON=jupyterexport PYSPARK_DRIVER_PYTHON_OPTS='notebook'
To save and exit the vi editor, press “ESC” and “:wq” to save it.
Restart your terminal and then type “pyspark”. You should be able to run jupyter notebook.
You should be able to see the port number in one of the lines.
Now to use the pmml library, open jupyter notebook by calling with the below command.
pyspark --jars /home/yashwanthmadaka_imp24/jpmml-sparkml-executable-1.5.3.jar
After opening the jupyter notebook, run all the cells which we had written earlier. Now, add the below piece of code.
trainingData = trainingData.drop(“features”)from pyspark.ml.feature import RFormulaformula = RFormula(formula = "label ~ .")classifier = DecisionTreeClassifier(maxDepth=25, minInstancesPerNode=30, impurity="gini")pipeline = Pipeline(stages = [formula, classifier])pipelineModel = pipeline.fit(trainingData)from pyspark2pmml import PMMLBuilderpmmlBuilder = PMMLBuilder(sc, trainingData, pipelineModel) \ .putOption(classifier, "compact", True)pmmlBuilder.buildFile("dt-stroke.pmml")
After running the above code, a new file will be created at the location mentioned. Download this file to your local desktop and let’s start building a website to interact with our model file.
The entire jupyter notebook can be found here.
Step 6a. Building a REST server from our model files:
For an application to interact with the model file, we need to expose the application as a REST web service. To do this, we are going to use the help of Openscoring library.
We need to install Openscoring using maven. Make sure you place the model file which we downloaded from our Google clouds VM machine into PATH/openscoring/openscoring-client/target folder.
where PATH = the path where the openscoring files exist.
After installation, we need to start the server by following the below commands.
First, start a server by going into the server folder and type the commands below
cd openscoring-server/targetjava -jar openscoring-server-executable-2.0-SNAPSHOT.jar
Next, start the client side by going to the client folder and type the below commands. Next, open a new cmd and type the below commands.
cd openscoring-client/targetjava -cp openscoring-client-executable-2.0-SNAPSHOT.jar org.openscoring.client.Deployer --model http://localhost:8080/openscoring/model/stroke --file dt-stroke.pmml
When we visit the http://localhost:8080/openscoring/model/stroke we can see the below structure
Step 6b. Downloading the ReactJS Frontend and running it:
Now visit the this Github link and clone this project.
After downloading, open this project folder using the VS Code. Open the terminal inside and type
npm install
Before starting our ReactJS application, we need to enable CORS. To do this, we can add a chrome extension.
After switching on the CORS, type the below command in VS Code terminal.
npm start
A web interface will open as shown below.
We can input any values and test it. We will get the prediction, the probability of occurrence of stroke and the probability of non-occurrence of stroke.
All the methods of interacting with the REST server are coded in the index.js file.
Reality model scores of 98 are impossible to achieve and the main significance of this blog is to show how to interact with ML models made in pyspark.
The better is our data preprocessing, the better will be our model. The quality of the model directly depends on the quality and diversity of the data we use. So, it’s better to spend more time doing proper data cleaning and data filtering techniques.
SMOTE — https://medium.com/coinmonks/smote-and-adasyn-handling-imbalanced-data-set-34f5223e167pyspark2pmml — https://github.com/jpmml/pyspark2pmmlOpenscoring — https://github.com/openscoring/openscoringReactJs Frontend — https://github.com/yashwanthmadaka24/React-Js-Website
SMOTE — https://medium.com/coinmonks/smote-and-adasyn-handling-imbalanced-data-set-34f5223e167
pyspark2pmml — https://github.com/jpmml/pyspark2pmml
Openscoring — https://github.com/openscoring/openscoring
ReactJs Frontend — https://github.com/yashwanthmadaka24/React-Js-Website
And now, its time for a break 😇