Deploying PySpark ML Model on Google Compute Engine as a REST API | by Ricky Kim | Towards Data Science

This post is loosely connected to my previous post.
Step-by-Step Tutorial: PySpark Sentiment Analysis on Google Dataproc
In my previous post, I trained a PySpark sentiment analysis model on Google Dataproc, and saved the model to Google Cloud Storage. In this post, I will show you how you can deploy a PySpark model on Google Compute Engine as a REST API. I will use the model I trained in my previous post, but I’m sure you can make some minor changes to the codes I will share and use with your own PySpark ML model.
I have the whole pipeline saved as a pipelineModel, and now I want to use the model for a REST API so that it can serve real-time predictions through simple REST API calls.
I first looked at Google Cloud ML Engine to see if this is a valid option for this specific use case. By reading through the introduction, you can not only train a machine learning model but also serve your model for predictions. But sadly it seems like Cloud ML Engine does not support Spark ML model. The next Google service I found was Google App Engine. This service made it really easy for anyone to deploy a web app. But after a couple of attempts, I realised it is not easy to set up Java Development Kit (which is needed to run PySpark) on the VM instances created through Google App Engine. It might be possible, but at least it was not straightforward enough for me to choose that route.
After some consideration and trials, below is the way that I found working. I will first go through the process step-by-step, then I will also tell you about the downside using Spark ML model for online real-time predictions.
In this tutorial, I will not go through the basic set up process again such as setting up a free account, enabling APIs for the services you want to use, installing Google Cloud SDK, but if this is your first time to try Google Cloud Platform I recommend you to check my previous post, and go through the set up steps (Creating a Free Trial Account on GCP, Enabling APIs, Installing Google Cloud SDK). Before you can proceed with the below steps, you will have to be ready by enabling Google Compute Engine API.
Now clone the git repository for this project by running below command in terminal.
git clone https://github.com/tthustla/flask_sparkml
Once you clone the repository, it will create a folder named flask_sparkml. Go into the folder and check what files are there.
cd flask_sparkml/ls
You will see three files and one subfolder.
install.sh (startup script when creating Compute Engine instance)
main.py (Flask web app to serve predictions as REST API)
model (the folder where trained PySpark pipeline model is stored)
response_time.py (simple Python script to measure API response time)
You can either create an instance through the web console or from your terminal using Google Cloud SDK. Before we create an instance, let’s take a quick look at the startup script install.sh to see what it does.
The above code will be a startup script for the VM.
Go to your console by visiting https://console.cloud.google.com/. Once you are in the console, click “Compute Engine” and “VM instances” from the left side menu. Click “Create”.
Type in the name for your VM instance, and choose the region and zone where you want your VM to be created. For this tutorial, we won’t use any other GCP services, so you can choose any region/zone you like, but it’s better to consider where the majority of the traffic will be coming from once the API is online.
Scroll down and choose “Allow full access to all Cloud APIs” in Identity and API access section, and tick “Allow HTTP traffic” in Firewall so that it can be reached from outside your VM. Finally, let’s add the startup script so that the VM will install required packages when starting up.
From the above screen just below Firewall section, you will see blue text “Management, security, disks, networking, sole tenancy”. Click on it to expand, and find the text box under “Startup script”. Copy and paste the whole install.sh code.
Click “Create” at the bottom.
In order to be able to interact with GCP from your terminal, you should be logged in to your account, and also it should be set to the project you intend to work on (I’m assuming you already have Google Cloud SDK installed on your terminal, if that’s not the case, please follow instructions on https://cloud.google.com/sdk/)
In order to check whether you’re signed in, copy and paste below command in your terminal, and it will show you which account is active.
gcloud auth list
If you want to check whether Google Cloud SDK is currently set to the project you want to work on, you can use below command from the terminal.
gcloud config list
If everything looks fine, run below command from your terminal in the flask_sparkml folder that you cloned from Git.
gcloud compute instances create flask-sparkml \--zone=europe-west1-b \--scopes=https://www.googleapis.com/auth/cloud-platform \--metadata-from-file startup-script=install.sh --tags http-server
This creates a VM instance with
instance name set to flask-sparkml
zone set to europe-west1-b
allow full access to all Cloud APIs
using local file install.sh as a startup script for the VM
allow HTTP traffic
In order to allow access to the application on port 8080 from outside, we need to create an inbound firewall rule which opens port 8080. Again you can choose to do this on web console or from the terminal.
From the left side menu on your console, find “VPC network” under “NETWORKING” section. Click into “Firewall rules” and click “CREATE FIREWALL RULE” button on the top. This will bring you to a screen looks like below.
First, give it a descriptive name. I named it “default-allow-http-8080”. Scroll down, and you will see the default setting of “Direction of traffic” is already set to “Ingress”, and also “Action on match” is set to “Allow”. For some reasons, if they are not, please make sure they are set properly.
Scroll down more to “Target tags” and give it a tag “http-server”. The tags are how GCP apply network rules to VM instances or instance template. When we created VM in the console, we have checked the tick box for “Allow HTTP traffic”. By doing this we are attaching any firewall rule with the tag “http-server” to the VM instance. Thus by creating a new firewall rule with the same tag will be automatically applied to your VM. Next, type in “0.0.0.0/0” for “Source IP ranges”. This will allow access to our API from any IP address. If you want to specify some limited ranges of IP, then you can do so here. Finally, under “Protocols and ports”, type in “8080” next to “tcp” and click “Create” button at the bottom.
If you want to create the firewall rule from your local terminal, you can run below command from your terminal.
gcloud compute firewall-rules create default-allow-http-8080 \  --allow tcp:8080 \  --source-ranges 0.0.0.0/0 \  --target-tags http-server
You can also check whether this firewall rule created properly.
gcloud compute firewall-rules list
Now a Compute Engine VM is created either through web console or from the terminal. To check whether the startup script is properly triggered, let’s connect to the VM. Again you can do this either through web console or from the terminal.
Go to “Compute Engine” and “VM instances” from the left side menu. You will see an instance running from the list. Click “SSH” under “Connect”.
If your default zone is set to the zone where your VM is created, and the project is set to the current project you are working on (properties you can see by running “gcloud config list” from the terminal), then you can simply type below command to SSH into your VM.
gcloud compute ssh [your_Google_user_name]@flask-sparkml
Now since we are on our VM, let’s inspect a few things. The startup script we provided earlier to VM includes Java JDK8 install and a few Python packages including PySpark. We can check whether they’re installed properly. (Please note that you need to give it a minute or two so that the VM will have enough time to finish the installations in the background.)
java -version
Java seems to be installed. What about the Python packages?
pip list
We can see four Python packages we specified in the startup script are installed on VM. Don’t close the VM terminal yet, since we will need to come back to this later.
The next step is to upload the main.py and model from our local machine to VM. Go to your local terminal and CD into flaks_sparkml (the folder cloned from Git) folder. Run below command to securely copy the files to VM.
gcloud compute scp --recurse . [your_Google_user_name]@flask-sparkml:~/flask_sparkml
Go back to VM terminal and check if the files are uploaded.
cd /home/[your_Google_user_name]/flask_sparkmlls -l
Finally, we are ready to run the actual Flask app.
Before we run the actual code, let’s take a quick look at the code to see what it does. The original code for this is from a Medium post Deploying a Machine Learning Model as a REST API by Nguyen Ngo, and I made some small changes to fit my specific use case. Thank you for the great tutorial Nguyen Ngo! I have already added comments inside the code, so I will not go through it line by line.
It’s time to run the application and make REST API call to the app to get a sentiment prediction from your own text! On your VM terminal (that you left open from the above “Connecting to Created VM Instance” step), run below command (using “nohup” command to prevent VM to go down after exiting VM terminal) from /home/[your_Google_user_name]/flask_sparkml directory.
nohup python main.py
You won’t be able to see the log output of the file main.py, since it writes logs to a file ‘nohup.out’. Let’s close the VM terminal to double check if ‘nohup’ is properly working, and get real-time sentiment prediction through an API call. In order to do that, we first need to know the external IP address that our app is deployed at. You can either check on your web console’s “Compute Engine” -> “VM Instances”
or check from your local terminal by running below command.
gcloud compute instances list
Either start Python from your local terminal or open a Jupyter Notebook and copy and paste below code block and run to see the result.
I expect to see the model to predict the above text as negative.
Hooray! The prediction from our model says it’s negative with quite high confidence.
You might have noticed that there is one more file in the folder that I haven’t mentioned yet. While going through this project, there has been one big question mark in my head. I know Spark can handle big data, and this might be beneficial in the model training stage. But how is the performance of Spark ML, when it is deployed to perform real-time prediction on single entries? When I made the API call in the above, it sure didn’t feel very fast. So I added a final experiment to the finished API. It is a simple Python program, designed to make 100 queries to the API and record API response times. At the end, it shows mean, median, min, max of this recorded response times.
Go to your local terminal, in the flask_sparkml folder, run the program and check the output.
python response_time.py http://[external_IP_address_of_your_app]:8080
Is this fast or slow? We have no way of knowing without a benchmark. So I have googled hard to find some other’s machine learning API response time. And I finally found this Medium post Falcon vs. Flask-Which one to pick to create a scalable deep learning REST API by Dat Tran. Thank you for the informative post, Dat Tran!
From Dat Tran’s post, the average response time for Keras CNN model on MNIST dataset is 60ms. We can see that the average response time of Spark ML is 0.99 seconds, which is 990ms. That doesn’t sound good at all.
While I was looking for an answer for this, I came across a presentation slide Productionizing Spark ML Pipelines with the Portable Format for Analytics by Nick Pentreath. Thank you for the great slide, Nick Pentreath! PFA (Portable Format for Analytics) is a JSON representation of Spark ML model that can be transferred across different languages, platforms. The reason why I’m mentioning this even when I didn’t export my model as PFA, is that he goes through some limitations of Spark ML model in production.
According to the slide (page 9: Challenges specific to Spark), scoring models in Spark is slow due to the overheads of Spark Dataframes and task scheduling, and not the best framework to be used for real-time scoring due to its latency.
Even though we have a working REST API deployed, this seems to be not the ideal solution to serve machine learning prediction real-time. I can explore further into exporting the model as PFA and compare the performance. Or I can also try to build a new model from scratch using Keras or Tensorflow, and deploy it to compare the performance with Spark ML. At the moment, I’m more tempted to try the latter, since it will give me chances to explore other GCP services such as Google Cloud ML Engine. Anyway, I will try to share the next part of my journey here.
To avoid incurring unwanted charges on your GCP account it’s important to clean up. You can simply delete the VM instance you created, but in case you want to know how to kill the nohup process we started with the file main.py, I will go through simple steps to kill processes running in the background. Go to your VM terminal either from web console or from your local terminal using “glcoud compute ssh” command. Once you are in the VM terminal, run below command to check the process ID of main.py running in the background.
ps aux | grep python | grep main.py
Once you have IDs of the currently running processes kill them using below command replacing [] parts with the process IDs of yours.
sudo kill [PID1] [PID2]
The actual deleting instance part can be done either on web console or from the terminal.
gcloud compute instances delete flask-sparkml
Thank you for reading. You can find the Git Repository of the scripts from the below link.