Title: Deep learning in production with Keras, Redis, Flask, and Apache - PyImageSearch

URL Source: https://pyimagesearch.com/2018/02/05/deep-learning-production-keras-redis-flask-apache/

Published Time: 2018-02-05T15:00:07+00:00

Markdown Content:
[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20343'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2018/01/deep_learning_cloud_animation.gif)

Shipping deep learning models to production is a non-trivial task.

If you don’t believe me, take a second and look at the “tech giants” such as Amazon, Google, Microsoft, etc. — _nearly all_ of them provide some method to ship your machine learning/deep learning models to production in the cloud.

Going with a model deployment service is perfectly fine and acceptable…**but what if you wanted to _own the entire process_ and _not rely on external services?_**

This type of situation is more common than you may think. Consider:

*   An in-house project where you cannot move sensitive data outside your network
*   A project that specifies that the entire infrastructure must reside within the company
*   A government organization that needs a private cloud
*   A startup that is in “stealth mode” and needs to stress test their service/application in-house

How would you go about shipping your deep learning models to production in these situations, and perhaps most importantly, **making it _scalable_ at the same time?**

Today’s post is the final chapter in our three part series on building a deep learning model server REST API:

1.  [Part one](https://blog.keras.io/building-a-simple-keras-deep-learning-rest-api.html) (**which was posted on the official Keras.io blog!**) is a simple Keras + deep learning REST API which is intended for single threaded use with no concurrent requests. This method is a perfect fit if this is your first time building a deep learning web server or if you’re working on a home/hobby project.
2.  In [part two](https://pyimagesearch.com/2018/01/29/scalable-keras-deep-learning-rest-api/) we demonstrated how to leverage **Redis** along with **message queueing/message brokering** paradigms to efficiently batch process incoming inference requests (but with a small caveat on server threading that could cause problems).
3.  In the final part of this series, I’ll show you how to resolve these server threading issues, further scale our method, provide benchmarks, and **demonstrate how to efficiently scale deep learning in production using Keras, Redis, Flask, and Apache.**

As the results of our stress test will demonstrate, our single GPU machine can easily handle **500 concurrent requests** (0.05 second delay in between each one) without ever breaking a sweat — _**this performance continues to scale as well.**_

**To learn how to ship your own deep learning models to production using Keras, Redis, Flask, and Apache, _just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

_**2020-06-16 Update:** This blog post is now TensorFlow 2+ compatible!_

The code for this blog post is primarily based on our [previous post](https://pyimagesearch.com/2018/01/29/scalable-keras-deep-learning-rest-api/), but with some minor modifications — the first part of today’s guide will review these changes along with our project structure.

From there, we’ll move on to configuring our deep learning web application, including installing and configuring any packages you may need (Redis, Apache, etc.).

Finally, we’ll stress test our server and benchmark our results.

For a quick overview of our deep learning production system (including a demo) be sure to watch the video above!

### Our deep learning project structure

Our project structure is as follows:

├── helpers.py
├── jemma.png
├── keras\_rest\_api\_app.wsgi
├── run\_model\_server.py
├── run\_web\_server.py
├── settings.py
├── simple\_request.py
└── stress\_test.py

Let’s review the important files:

*   `run_web_server.py` contains all our Flask web server code — Apache will load this when starting our deep learning web app.
*   `run_model_server.py` will:
    *   Load our Keras model from disk
    *   Continually poll Redis for new images to classify
    *   Classify images (batch processing them for efficiency)
    *   Write the inference results back to Redis so they can be returned to the client via Flask
*   `settings.py` contains all Python-based settings for our deep learning productions service, such as Redis host/port information, image classification settings, image queue name, etc.
*   `helpers.py` contains utility functions that both `run_web_server.py` and `run_model_server.py` will use (namely `base64` encoding).
*   `keras_rest_api_app.wsgi` contains our WSGI settings so we can serve the Flask app from our Apache server.
*   `simple_request.py` can be used to programmatically consume the results of our deep learning API service.
*   `jemma.png` is a photo of my family’s beagle. We’ll be using her as an example image when calling the REST API to validate it is indeed working.
*   Finally, we’ll use `stress_test.py` to stress our server and measure image classification throughout.

As described [last week](https://pyimagesearch.com/2018/01/29/scalable-keras-deep-learning-rest-api/), we have a single endpoint on our Flask server, `/predict` . This method lives in `run_web_server.py` and will compute the classification for an input image on demand. Image pre-processing is also handled in `run_web_server.py` .

In order to make our server production-ready, I’ve pulled out the `classify_process` function from last week’s single script and placed it in `run_model_server.py` . This script is very important as it will load our Keras model and grab images from our image queue in Redis for classification. Results are written back to Redis (the `/predict` endpoint and corresponding function in `run_web_server.py` monitors Redis for results to send back to the client).

**But what good is a deep learning REST API server unless we know its capabilities and limitations?**

In `stress_test.py` , we test our server. We’ll accomplish this by kicking off 500 concurrent threads which will send our images to the server for classification in parallel. I recommend running this on the server localhost to start, and then running it from a client that is off site.

### Building our deep learning web app

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20340'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2018/01/keras_api_header.png)

**Figure 1:** Data flow diagram for a deep learning REST API server built with Python, Keras, Redis, and Flask.

Nearly every single line of code used in this project comes from our previous post on building a [scalable deep learning REST API](https://pyimagesearch.com/2018/01/29/scalable-keras-deep-learning-rest-api/) — **the only change is that we are moving some of the code to _separate files_ to facilitate scalability in a production environment.**

As a matter of completeness I’ll be including the source code to each file in this blog post (and in the _**“Downloads”**_ section of this blog post). **For a detailed review of the files, please [see the previous post](https://pyimagesearch.com/2018/01/29/scalable-keras-deep-learning-rest-api/).**

#### Settings and configurations

\# initialize Redis connection settings
REDIS\_HOST = "localhost"
REDIS\_PORT = 6379
REDIS\_DB = 0

# initialize constants used to control image spatial dimensions and
# data type
IMAGE\_WIDTH = 224
IMAGE\_HEIGHT = 224
IMAGE\_CHANS = 3
IMAGE\_DTYPE = "float32"

# initialize constants used for server queuing
IMAGE\_QUEUE = "image\_queue"
BATCH\_SIZE = 32
SERVER\_SLEEP = 0.25
CLIENT\_SLEEP = 0.25

In `settings.py` you’ll be able to change parameters for the server connectivity, image dimensions + data type, and server queuing.

#### Helper utilities

\# import the necessary packages
import numpy as np
import base64
import sys

def base64\_encode\_image(a):
	# base64 encode the input NumPy array
	return base64.b64encode(a).decode("utf-8")

def base64\_decode\_image(a, dtype, shape):
	# if this is Python 3, we need the extra step of encoding the
	# serialized NumPy string as a byte object
	if sys.version\_info.major == 3:
		a = bytes(a, encoding="utf-8")

	# convert the string to a NumPy array using the supplied data
	# type and target shape
	a = np.frombuffer(base64.decodestring(a), dtype=dtype)
	a = a.reshape(shape)

	# return the decoded image
	return a

The `helpers.py` file contains two functions — one for `base64` _encoding_ and the other for _decoding_.

Encoding is necessary so that we can serialize + store our image in Redis. Likewise, decoding is necessary so that we can deserialize the image into NumPy array format prior to pre-processing.

#### The deep learning web server

\# import the necessary packages
from tensorflow.keras.preprocessing.image import img\_to\_array
from tensorflow.keras.applications.resnet50 import preprocess\_input
from PIL import Image
import numpy as np
import settings
import helpers
import flask
import redis
import uuid
import time
import json
import io

# initialize our Flask application and Redis server
app = flask.Flask(\_\_name\_\_)
db = redis.StrictRedis(host=settings.REDIS\_HOST,
	port=settings.REDIS\_PORT, db=settings.REDIS\_DB)

def prepare\_image(image, target):
	# if the image mode is not RGB, convert it
	if image.mode != "RGB":
		image = image.convert("RGB")

	# resize the input image and preprocess it
	image = image.resize(target)
	image = img\_to\_array(image)
	image = np.expand\_dims(image, axis=0)
	image = preprocess\_input(image)

	# return the processed image
	return image

@app.route("/")
def homepage():
	return "Welcome to the PyImageSearch Keras REST API!"

@app.route("/predict", methods=\["POST"\])
def predict():
	# initialize the data dictionary that will be returned from the
	# view
	data = {"success": False}

	# ensure an image was properly uploaded to our endpoint
	if flask.request.method == "POST":
		if flask.request.files.get("image"):
			# read the image in PIL format and prepare it for
			# classification
			image = flask.request.files\["image"\].read()
			image = Image.open(io.BytesIO(image))
			image = prepare\_image(image,
				(settings.IMAGE\_WIDTH, settings.IMAGE\_HEIGHT))

			# ensure our NumPy array is C-contiguous as well,
			# otherwise we won't be able to serialize it
			image = image.copy(order="C")

			# generate an ID for the classification then add the
			# classification ID + image to the queue
			k = str(uuid.uuid4())
			image = helpers.base64\_encode\_image(image)
			d = {"id": k, "image": image}
			db.rpush(settings.IMAGE\_QUEUE, json.dumps(d))

			# keep looping until our model server returns the output
			# predictions
			while True:
				# attempt to grab the output predictions
				output = db.get(k)

				# check to see if our model has classified the input
				# image
				if output is not None:
					# add the output predictions to our data
					# dictionary so we can return it to the client
					output = output.decode("utf-8")
					data\["predictions"\] = json.loads(output)

					# delete the result from the database and break
					# from the polling loop
					db.delete(k)
					break

				# sleep for a small amount to give the model a chance
				# to classify the input image
				time.sleep(settings.CLIENT\_SLEEP)

			# indicate that the request was a success
			data\["success"\] = True

	# return the data dictionary as a JSON response
	return flask.jsonify(data)

# for debugging purposes, it's helpful to start the Flask testing
# server (don't use this for production
if \_\_name\_\_ == "\_\_main\_\_":
	print("\* Starting web service...")
	app.run()

Here in `run_web_server.py` , you’ll see `predict` , the function associated with our REST API `/predict` endpoint.

The `predict` function pushes the encoded image into the Redis queue and then continually loops/polls until it obains the prediction data back from the model server. We then JSON-encode the data and instruct Flask to send the data back to the client.

#### The deep learning model server

\# import the necessary packages
from tensorflow.keras.applications import ResNet50
from tensorflow.keras.applications.resnet50 import decode\_predictions
import numpy as np
import settings
import helpers
import redis
import time
import json

# connect to Redis server
db = redis.StrictRedis(host=settings.REDIS\_HOST,
	port=settings.REDIS\_PORT, db=settings.REDIS\_DB)

def classify\_process():
	# load the pre-trained Keras model (here we are using a model
	# pre-trained on ImageNet and provided by Keras, but you can
	# substitute in your own networks just as easily)
	print("\* Loading model...")
	model = ResNet50(weights="imagenet")
	print("\* Model loaded")

	# continually pool for new images to classify
	while True:
		# attempt to grab a batch of images from the database, then
		# initialize the image IDs and batch of images themselves
		queue = db.lrange(settings.IMAGE\_QUEUE, 0,
			settings.BATCH\_SIZE - 1)
		imageIDs = \[\]
		batch = None

		# loop over the queue
		for q in queue:
			# deserialize the object and obtain the input image
			q = json.loads(q.decode("utf-8"))
			image = helpers.base64\_decode\_image(q\["image"\],
				settings.IMAGE\_DTYPE,
				(1, settings.IMAGE\_HEIGHT, settings.IMAGE\_WIDTH,
					settings.IMAGE\_CHANS))

			# check to see if the batch list is None
			if batch is None:
				batch = image

			# otherwise, stack the data
			else:
				batch = np.vstack(\[batch, image\])

			# update the list of image IDs
			imageIDs.append(q\["id"\])

		# check to see if we need to process the batch
		if len(imageIDs) > 0:
			# classify the batch
			print("\* Batch size: {}".format(batch.shape))
			preds = model.predict(batch)
			results = decode\_predictions(preds)

			# loop over the image IDs and their corresponding set of
			# results from our model
			for (imageID, resultSet) in zip(imageIDs, results):
				# initialize the list of output predictions
				output = \[\]

				# loop over the results and add them to the list of
				# output predictions
				for (imagenetID, label, prob) in resultSet:
					r = {"label": label, "probability": float(prob)}
					output.append(r)

				# store the output predictions in the database, using
				# the image ID as the key so we can fetch the results
				db.set(imageID, json.dumps(output))

			# remove the set of images from our queue
			db.ltrim(settings.IMAGE\_QUEUE, len(imageIDs), -1)

		# sleep for a small amount
		time.sleep(settings.SERVER\_SLEEP)

# if this is the main thread of execution start the model server
# process
if \_\_name\_\_ == "\_\_main\_\_":
	classify\_process()

The `run_model_server.py` file houses our `classify_process` function. This function loads our model and then runs predictions on a batch of images. This process is ideally excuted on a GPU, but a CPU can also be used.

In this example, for sake of simplicity, we’ll be using ResNet50 pre-trained on the ImageNet dataset. You can modify `classify_process` to utilize your own deep learning models.

#### The WSGI configuration

\# add our app to the system path
import sys
sys.path.insert(0, "/var/www/html/keras-complete-rest-api")

# import the application and away we go...
from run\_web\_server import app as application

Our next file, `keras_rest_api_app.wsgi` , is a new component to our deep learning REST API compared to last week.

This WSGI configuration file adds our server directory to the system path and imports the web app to kick off all the action. We point to this file in our Apache server settings file, `/etc/apache2/sites-available/000-default.conf` , later in this blog post.

#### The stress test

\# import the necessary packages
from threading import Thread
import requests
import time

# initialize the Keras REST API endpoint URL along with the input
# image path
KERAS\_REST\_API\_URL = "http://localhost/predict"
IMAGE\_PATH = "jemma.png"

# initialize the number of requests for the stress test along with
# the sleep amount between requests
NUM\_REQUESTS = 500
SLEEP\_COUNT = 0.05

def call\_predict\_endpoint(n):
	# load the input image and construct the payload for the request
	image = open(IMAGE\_PATH, "rb").read()
	payload = {"image": image}

	# submit the request
	r = requests.post(KERAS\_REST\_API\_URL, files=payload).json()

	# ensure the request was sucessful
	if r\["success"\]:
		print("\[INFO\] thread {} OK".format(n))

	# otherwise, the request failed
	else:
		print("\[INFO\] thread {} FAILED".format(n))

# loop over the number of threads
for i in range(0, NUM\_REQUESTS):
	# start a new thread to call the API
	t = Thread(target=call\_predict\_endpoint, args=(i,))
	t.daemon = True
	t.start()
	time.sleep(SLEEP\_COUNT)

# insert a long sleep so we can wait until the server is finished
# processing the images
time.sleep(300)

Our `stress_test.py` script will help us to test the server and determine its limitations. I always recommend stress testing your deep learning REST API server so that you know _if_ (and more importantly, _when_) you need to add additional GPUs, CPUs, or RAM. This script kicks off `NUM_REQUESTS` threads and POSTs to the `/predict` endpoint. It’s up to our Flask web app from there.

### Configuring our deep learning production environment

This section will discuss how to install and configure the necessary prerequisites for our deep learning API server.

We’ll use my [PyImageSearch Deep Learning AMI](https://pyimagesearch.com/2017/09/20/pre-configured-amazon-aws-deep-learning-ami-with-python/) (freely available to you to use) as a base. I chose a **p2.xlarge** instance with a single GPU for this example.

You can modify the code in this example to leverage multiple GPUs as well by:

1.  Running multiple model server processes
2.  Maintaining an image queue for _each_ GPU and corresponding model process

However, keep in mind that your machine will still be limited by I/O. It may be beneficial to instead utilize multiple machines, each with 1-4 GPUs than trying to scale to 8 or 16 GPUs on a single machine.

#### Compile and installing Redis

[Redis](https://redis.io/), an efficient in-memory database, will act as our queue/message broker.

Obtaining and installing Redis is very easy:

$ wget http://download.redis.io/redis-stable.tar.gz
$ tar xvzf redis-stable.tar.gz
$ cd redis-stable
$ make
$ sudo make install

#### Create your deep learning Python virtual environment

Be sure to install `virtualenv` and `virtualenvwrapper` to manage Python virtual environments on your system. You can follow either of these guides:

*   [_How to install TensorFlow 2.0 on Ubuntu_](https://pyimagesearch.com/2019/12/09/how-to-install-tensorflow-2-0-on-ubuntu/)
*   [_How to install TensorFlow 2.0 on macOS_](https://pyimagesearch.com/2019/12/09/how-to-install-tensorflow-2-0-on-macos/)

Please note that [PyImageSearch does not recommend or support Windows for CV/DL projects](https://pyimagesearch.com/faqs/single-faq/can-you-help-me-do-___-on-windows/).

After following those instructions, you’ll have a Python 3 virtual environment named `dl4cv`.

From there, you’ll need the following additional packages:

$ workon dl4cv
$ pip install flask
$ pip install gevent
$ pip install requests
$ pip install redis

#### Install the Apache web server

Other web servers can be used such as [nginx](https://www.nginx.com/) but since I have more experience with Apache (and therefore more familiar with Apache in general), I’ll be using Apache for this example.

Apache can be installed via:

$ sudo apt-get install apache2

If you’ve created a virtual environment using **Python 3** you’ll want to install the Python 3 WSGI + Apache module:

$ sudo apt-get install libapache2-mod-wsgi-py3
$ sudo a2enmod wsgi

To validate that Apache is installed, open up a browser and enter the IP address of your web server. If you can’t see the server splash screen then be sure to open up Port 80 and Port 5000.

In my case, the IP address of my server is `54.187.46.215` (yours will be different). Entering this in a browser I see:

[![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20344'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2018/01/deep_learning_production_apache.jpg)

**Figure 2:** The default Apache splash screen lets us know that Apache is installed and that it can be accessed from an open port 80.

…which is the default Apache homepage.

#### Sym-link your Flask + deep learning app

By default, Apache serves content from `/var/www/html` . I would recommend creating a _sym-link_ from `/var/www/html` to your Flask web app.

I have uploaded my deep learning + Flask app to my home directory in a directory named `keras-complete-rest-api` :

$ ls ~
keras-complete-rest-api

I can sym-link it to `/var/www/html` via:

$ cd /var/www/html/
$ sudo ln -s ~/keras-complete-rest-api keras-complete-rest-api

### Update your Apache configuration to point to the Flask app

In order to configure Apache to point to our Flask app, we need to edit `/etc/apache2/sites-available/000-default.conf` .

Open in your favorite text editor (here I’ll be using `vi` ):

$ sudo vi /etc/apache2/sites-available/000-default.conf

At the top of the file supply your `WSGIPythonHome` (path to Python `bin` directory) and `WSGIPythonPath` (path to Python `site-packages` directory) configurations:

WSGIPythonHome /home/ubuntu/.virtualenvs/keras\_flask/bin
WSGIPythonPath /home/ubuntu/.virtualenvs/keras\_flask/lib/python3.5/site-packages

<VirtualHost \*:80>
	...
</VirtualHost>

_**2020-06-18 Update:** On Ubuntu 18.04, you may need to change the first line to:_

`WSGIPythonHome /home/ubuntu/.virtualenvs/keras_flask`

_… where `/bin` is eliminated._

Since we are using Python virtual environments in this example (I have named my virtual environment `keras_flask` ), we supply the path to the `bin` and `site-packages` directory for the Python virtual environment.

Then in body of `<VirtualHost>` , right after `ServerAdmin` and `DocumentRoot` , add:

<VirtualHost \*:80>
	...
	
	WSGIDaemonProcess keras\_rest\_api\_app threads=10
	WSGIScriptAlias / /var/www/html/keras-complete-rest-api/keras\_rest\_api\_app.wsgi
	
	<Directory /var/www/html/keras-complete-rest-api>
		WSGIProcessGroup keras\_rest\_api\_app
		WSGIApplicationGroup %{GLOBAL}
		Order deny,allow
		Allow from all
	</Directory>
	
	...
</VirtualHost>

#### Sym-link CUDA libraries (optional, GPU only)

If you’re using your GPU for deep learning and want to leverage CUDA (and why wouldn’t you), Apache unfortunately has no knowledge of CUDA’s `*.so` libraries in `/usr/local/cuda/lib64` .

I’m not sure what the “most correct” way instruct to Apache of where these CUDA libraries live, but the “total hack” solution is to sym-link _all_ files from `/usr/local/cuda/lib64` to `/usr/lib` :

$ cd /usr/lib
$ sudo ln -s /usr/local/cuda/lib64/\* ./

If there is a better way to make Apache aware of the CUDA libraries, _please let me know in the comments._

#### Restart the Apache web server

Once you’ve edited your Apache configuration file and optionally sym-linked the CUDA deep learning libraries, be sure to restart your Apache server via:

$ sudo service apache2 restart

#### Testing your Apache web server + deep learning endpoint

To test that Apache is properly configured to deliver your Flask + deep learning app, refresh your web browser:

[![Image 5](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20302'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2018/01/deep_learning_production_apache_configured.png)

**Figure 3:** Apache + Flask have been configured to work and I see my welcome message.

You should now see the text _“Welcome to the PyImageSearch Keras REST API!”_ in your browser.

Once you’ve reached this stage your Flask deep learning app should be ready to go.

All that said, if you run into any problems make sure you refer to the next section…

#### TIP: Monitor your Apache error logs if you run into trouble

I’ve been using Python + web frameworks such as Flask and Django for _years_ and I still make mistakes when getting my environment configured properly.

While I wish there was a bullet proof way to make sure everything works out of the gate, the truth is something is likely going to gum up the works along the way.

The good news is that WSGI logs Python events, including failures, to the server log.

On Ubuntu, the Apache server log is located in `/var/log/apache2/` :

$ ls /var/log/apache2
access.log error.log other\_vhosts\_access.log

When debugging, I often keep a terminal open that runs:

$ tail -f /var/log/apache2/error.log

…so I can see the _second_ an error rolls in.

Use the error log to help you get Flask up and running on your server.

### Starting your deep learning model server

Your Apache server _should_ already be running. If not, you can start it via:

$ sudo service apache2 start

You’ll then want to start the Redis store:

$ redis-server

And in a separate terminal launch the Keras model server:

$ python run\_model\_server.py
\* Loading model...
...
\* Model loaded

From there try to submit an example image to your deep learning API service:

$ curl -X POST -F image=@jemma.png 'http://localhost/predict'
{
  "predictions": \[
    {
      "label": "beagle", 
      "probability": 0.9461532831192017
    }, 
    {
      "label": "bluetick", 
      "probability": 0.031958963721990585
    }, 
    {
      "label": "redbone", 
      "probability": 0.0066171870566904545
    }, 
    {
      "label": "Walker\_hound", 
      "probability": 0.003387963864952326
    }, 
    {
      "label": "Greater\_Swiss\_Mountain\_dog", 
      "probability": 0.0025766845792531967
    }
  \], 
  "success": true
}

If everything is working, you should receive formatted JSON output back from the deep learning API model server with the class predictions + probabilities.

[![Image 6](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20377'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2018/01/keras_api_result01.jpg)

**Figure 4:** Using cURL to test our Keras REST API server. Pictured is my family beagle, Jemma. She is classified as a beagle with 94.6% confidence by our ResNet model.

### Stress testing your deep learning REST API

Of course, this is just an example. Let’s stress test our deep learning REST API.

Open up another terminal and execute the following command:

$ python stress\_test.py 
\[INFO\] thread 3 OK
\[INFO\] thread 0 OK
\[INFO\] thread 1 OK
...
\[INFO\] thread 497 OK
\[INFO\] thread 499 OK
\[INFO\] thread 498 OK

In your `run_model_server.py` output you’ll start to see the following lines logged to the terminal:

\* Batch size: (4, 224, 224, 3)
\* Batch size: (9, 224, 224, 3)
\* Batch size: (9, 224, 224, 3)
\* Batch size: (8, 224, 224, 3)
...
\* Batch size: (2, 224, 224, 3)
\* Batch size: (10, 224, 224, 3)
\* Batch size: (7, 224, 224, 3)

Even with a new request coming in every 0.05 seconds our batch size never gets larger than ~10-12 images per batch.

Our model server handles the load easily without breaking a sweat and it can easily scale beyond this.

If you do overload the server (perhaps your batch size is too big and you run out of GPU memory with an error message), you should stop the server, and use the Redis CLI to clear the queue:

$ redis-cli
> FLUSHALL

From there you can adjust settings in `settings.py` and `/etc/apache2/sites-available/000-default.conf` . Then you may restart the server.

For a full demo, please see the video below:

### Recommendations for deploying your own deep learning models to production

One of the best pieces of advice I can give is to keep your data, in particular your Redis server, close to the GPU.

You _may_ be tempted to spin up a giant Redis server with hundreds of gigabytes of RAM to handle multiple image queues and serve multiple GPU machines.

The problem here will be I/O latency and network overhead.

Assuming _224 x 224 x 3_ images represented as float32 array, a batch size of 32 images will be ~19MB of data. This implies that for each batch request from a model server, Redis will need to pull out 19MB of data and send it to the server.

On fast switches this isn’t a big deal, but you should consider running _both_ your model server _and_ Redis on the same server to keep your data close to the GPU.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

Summary
-------

In today’s blog post we learned how to deploy a deep learning model to production using Keras, Redis, Flask, and Apache.

Most of the tools we used here are interchangeable. You could swap in TensorFlow or PyTorch for Keras. Django could be used instead of Flask. Nginx could be swapped in for Apache.

The only tool I would _not_ recommend swapping out is Redis. Redis is arguably the best solution for in-memory data stores. Unless you have a specific reason to not use Redis, I would suggest utilizing Redis for your queuing operations.

Finally, we stress tested our deep learning REST API.

We submitted a total of 500 requests for image classification to our server with 0.05 second delays in between each — our server was not phased (the batch size for the CNN was never more than ~37% full).

Furthermore, this method is easily scalable to additional servers. If you place these servers behind a load balancer you can easily scale this method further.

I hope you enjoyed today’s blog post!

**To be notified when future blog posts are published on PyImageSearch, _be sure to enter your email address in the form below!_**

![Image 7](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!