How to Deploy Keras Models to Production (Beginners Welcome!)
By: Zain Raza
Tags: Keras, DevOps, Flask, AWS, Json

Photo by Taylor Vick on Unsplash

Lessons Learned in Deploying Neural Networks

Let me begin by saying I have a really exciting job — but this blog is not going to be about the exciting parts.

I want to speak to you about the problems you may face in deploying models implemented in Keras, the popular open source framework for deep learning. This is something many engineers (like myself) have struggled with, and you may spend hours trying to figure this out if you’re on your own. My objective though is to make sure you don’t need to do that, and that you’ll come away from this blog with 3 Learning Outcomes:

Implement RESTful APIs in warp speed for your Deep Learning model using the Flask-RESTPlus framework Use the AWS SDK for Python, to Provide Cloud-Stored Data to the API Parse the JSON Responses from the API, so you can use it in your app!

The good news is if I do my job right, you’ll see by the end that it’s actually very approachable.

Lesson #1: Separate Concerns Between Machine Learning, and The HTTP Routes

As you read this, I assume a few things about your deep learning project:

You already have a trained model, and have saved it. NOTE: the examples in this blog will use a Hadoop (.h5) and JSON (.json) file, so please keep that in mind if you are using a different format. You are deploying the model to a web server. Unfortunately I cannot promise that these concepts will carry over to mobile or IoT devices (see TFLite), or the browser (see Tensorflow.js).

If both those statements apply to you though, then I can sum up this lesson in 4 seconds:

Let Someone Else Worry About the Model

That can seem confusing at first, however this lesson is really just about the fundamentals — have a dedicated API endpoint for running inference, and I promise the deployment process becomes a lot simpler.

Example: Building a RESTful API with Flask-RESTPlus

As you may know, the simplest way you can load your saved models in Keras to use on a web server is by simply providing the relative file paths on your local machine[1].

For example, let’s take a look at Plant Vision, a RESTful API I built using Flask-RESTPlus. The code for the repository is linked here, and it already comes with a saved model (to use for image classification) so feel free to play around with it as you wish.

For context, this API allows users to upload images of plant leaves, and the model (a convolutional neural network) identifies them as healthy or not.

We start by importing the packages we need to program a web server in Flask, build a RESTful API with Flask-RESTPlus, and to process our image data using Keras with a Tensorflow backend:

# Prevent ImportErrors w/ flask

import werkzeug

werkzeug.cached_property = werkzeug.utils.cached_property

from werkzeug.datastructures import FileStorage

# ML/Data processing

import tensorflow.keras as keras

# RESTful API packages

from flask_restplus import Api, Resource

from flask import Flask

# Local utility Functions

from util import leaf

2. We’ll need to initialize an app variable for Flask of course:

app = Flask(__name__)

3. The next step is to build the API, and this is where the usefulness of Flask-RESTPlus shows. In this next snippet, we’ll instantiate an API object, and give it a namespace — you’ll see what I mean below:

api = Api(app, version="1.0", title="PlantNet API",

description="Identifying Leaf Disease via Deep Learning") ns = api.namespace("Diagnosis", description="Represents the health states of a leaf understood by the AI."

)

You can run the Flask server in development now, using the following command (this is specific to the repo linked above):

export FLASK_ENV=development; export FLASK_APP=application; flask run

And see we already have something to show users!

Auto-generated UI for a Flask-RESTPlus API

Look back up at the previous code snippet — notice any similarities between the keyword arguments we provided, and the text that appears on the screenshot above?

Normally in Flask we would need to make separate HTML templates to display a UI to users.

But because you’re a smartypants, you now know that by using Flask-RESTPlus, we can auto-generate a UI just based on the keyword arguments we provided to the Api and api.namespace constructors!

the version you provide Api becomes that little number 1 ;

you provide becomes that little number ; the title you provided as well becomes the header of the template;

you provided as well becomes the header of the template; and the argument provided for description subtitles the page

But wait a second —what does it mean by “No operations defined in spec”? And how come it looks like nothing from the namespace appears on the template?

This gets us into the next step: the purpose of the namespaces in Flask-RESTPlus is to group your API routes together.

So until we actually have added a route for our model to classify images, there’s really isn’t a need for anything else on the UI (because we haven’t really built the API yet).

3. Let’s fix that now —let’s create a route that uses the model to infer the health of user-uploaded leaf images. Use the following code snippet to load the deep learning model as a global object, and implement this route:

# Use Flask-RESTPlus argparser to process user-uploaded images

arg_parser = api.parser()

arg_parser.add_argument('image', location='files',

type=FileStorage, required=True) # Model reconstruction - using retrained weights from Inception V3

with open("./plantnet/inceptionModelArchitecture.json") as f:

model = keras.models.model_from_json(f.read())

model.load_weights("./plantnet/inception_model_weights.h5") # Add the route to run inference

@ns.route("/prediction")

class CNNPrediction(Resource):

"""Takes in the image, to pass to the CNN"""

@api.doc(parser=arg_parser,

description="Let the AI predict if a leaf is healthy.")

def post(self):

# A: get the image

image = leaf.get_image(arg_parser)

# B: preprocess the image

final_image = leaf.preprocess_image(image)

# C: make the prediction

prediction = leaf.predict_leaf_health(model, final_image)

# return the classification

return prediction

If you want to go into more detail on how the model actually preprocesses images and predicts on them, take a detour to the utility functions in the repository.

Now when you reload the tab where you’re viewing the auto-generated UI, you should now see the API namespace show up:

Once we have added routes, the namespace that groups them shows up as well.

In fact you can even drop down the namespace to reveal a debugger, so you can test out the route we’ve created (using the sample image in the repo for example), and check to make sure it works like you’d expect:

Again, let’s emphasize that this route’s only purpose is to provide a label for one image of a leaf — nothing more, nothing less. A sample input image may look like the following:

And the output is a JSON string such as the following:

{

"label": "Grape_Black rot",

"confidence": 0.4162355065345764

}

Key Takeaways

In this example, we’ve seen that can reduce the complexity of deploying deep learning models to the web, by encapsulating all the data processing into a API endpoint (and do it quickly with a framework like Flask-RESTPlus).

The Next Steps

For now, let’s move beyond the debugging UI— the real magic comes in using the API endpoint we’ve created in the backend of other web apps we may have. So let’s take a look at that now, and see how easy it becomes to use deep learning in your own projects!

“Inside every large program is a small program struggling to get out.”

― Tony Hoare, 1980 ACM Turing Award Recipient

Lesson #2: Don’t Let Cloud Storage Stop You

Truthfully speaking, you’ve already done a huge chunk of the work my friend. There’s just a few things left that can trip us up in using our API — like, how are you supposed to send images programmatically in an HTTP request?

Again, this example has a few assumptions:

If your app stores the images in the cloud (specifically, in an AWS S3 bucket that has programmatic access)[3]. Your app’s backend is written in Python.

If those assumptions do apply, then this next example is just for you!

Example: Accessing S3 Objects Using boto3

Let’s say we have a web app for botanists who want to upload leaf images of the plants they study. We want our app to use the Plant Vision API described above, to let them know if the leaves are healthy.

We can use boto3 , the AWS SDK for Python, to get images stored from our S3 bucket (more generally known as S3 “objects”) into our program.

For starters, let’s make a .env in the repository of our other app, and add the AWS credentials we need to access the S3 bucket:

AWS_ACCESS_KEY_ID=<your_access_key>

AWS_SECRET_ACCESS_KEY=<your_secret_key>

AWS_STORAGE_BUCKET_NAME=<name_of_your_bucket>

2. Next, let’s make sure we can use these environment variables in our application. There’s a few ways to do this in Python, for this example we’ll use the python-dotenv package, to be installed via pip:

(env) python3 -m pip install python-dotenv

Then, we can load in our environment variables to our Python module like so:

# Python module

from dotenv import load_dotenv

# Loads in environment variables from a .env file

load_dotenv()

# Examples of Storing AWS information in Python variables

# AWS S3 Variables

AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID", "")

AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY", "")

AWS_STORAGE_BUCKET_NAME = os.getenv("AWS_STORAGE_BUCKET_NAME", "")

3. Now, we’re all set to actually access the S3 objects!

We start by initializing the AWS client — this will let us interface with our S3 bucket:

# init AWS client

import boto3

s3 = boto3.resource('s3')

bucket = s3.Bucket(settings.AWS_STORAGE_BUCKET_NAME)

4. In order to get the S3 object, we now need it’s key. This step is very tactical — for example, let’s say the bucket has an image stored in it using the following structure:

*** My S3 Bucket File Directory ***

garden-images/

|

|

|---plantA/

|

|

my-plant-img.jpg

Then the key for my-plant-img.jpg would be garden-images/plantA/my-plant-img.jpg .

However, your app’s database might only store the HTTPS address to the image stored in S3, so there’s a ton of extra information that would make boto3 throw an error!

To fix this, we can convert your image’s URL address (which may be using HTTPS) to an object key on S3, by slicing the URL to take only the root directory of the bucket, to the start of the query string:

# get the key to the image on S3, leaving out the rest

ROOT_BUCKET_DIR = "garden-images"

start_path = img_url.find(ROOT_BUCKET_DIR)

end_path = img_url.find("?")

key = img_url[start_path:end_path]

Note that for this work, your img_url should start as a URL address! You will also need to study the file structure of your own S3 bucket, to find the ROOT_BUCKET_DIR you will need.

4. We can now grab the image data stored on S3 with the following:

# get the image data

object = bucket.Object(key)

response = object.get()['Body']

5. With that, you’re now all set to send this image in a HTTP request to the API, by using the requests module:

# convert the image to low-level binary data

img_data = response.read()

img_bytes = bytearray(img_data)

# data needed to make the request

files = {'image': img_bytes}

url = "https://plantvision.herokuapp.com/Diagnosis/prediction"\

# get the predictions from the Plant Vision API

api_response = requests.post(url, files=files)

NOTES:

The files dictionary you make has to be structured for the specific API you’re using. The only reason I used ‘image’ as they key in this dictionary is because that is the argument we passed to arg_parser in the Flask-RESTPlus (see the example in the previous lesson). As a Python programmer, we don’t normally see the bytearray data structure — so what is it, you may ask? It’s essentially what we can use to transfer the image information over the TCP/IP network to our API, as it needs to be low-level bytes[2].

6. Last but not least, we need to make sure we actually use the data! Right now our API returns a dictionary, which goes inside of our response.text .

We probably don’t need all the data in the response variable. Let’s use the json package to parse the response from our API, storing the values that we care about in their own variables for further use:

import json

# parse the response

label = json.loads(response.text)['label']

confidence = json.loads(response.text)['confidence']

Key Takeaways

In this section, we saw how to make requests involving image data to a RESTful API in Python. We overcame the issues of accessing the images in AWS by using boto3 and a little string parsing, and made the HTTP request by using a bytearray in the dictionary we eventually pass to requests.post .

I realize this part has a lot more steps to it, so here’s the link to the code on GitHub for the other app I’m using in the code snippets.

“The cloud is for everyone. The cloud is a democracy.” –Marc Benioff, CEO of Salesforce

Closing Thoughts

For now, we’ve gotten a response from the deep learning model into our app. However, is this enough to actually provide value to users?

At Carbon0 Games, my team and I believe we can advance how humans think, feel, and act in designing their lives to fight climate change. If deep learning can help us in that mission, that’s great — however it’s not the end of the story.

If you’ve made it this far, I expect you too will try to deploy deep learning as a force for positive change. Build things your users can trust, help more of us get through these unprecedented times— because that’s a job truly worth getting excited about.

References and Further Reading

Please check out some of the following resource if you would like to know more on some of the topics mentioned here:

[1] Kathy Wu, Francois Chollet, “Serialization and saving” (2020), Keras.io.

[2] Jakub Przywóski, “bytearray”, (2015), Python Reference (The Right Way).

[3] Corey Schafer, “Python Django Tutorial: Full-Featured Web App Part 13 — Using AWS S3 for File Uploads” (2019), YouTube.

And lastly, if you want to build more robust production pipelines I’d recommend the Tensorflow Serving documentation. Alternatively, this blog post by an Amazon Data Scientist explains how to use AWS Sagemaker if you want to just run model inference in the cloud.