There are two very different ways to deploy ML models, here’s both | by Tom Grek | Towards Data Science

If an ML model makes a prediction in Jupyter, is anyone around to hear it?
Probably not. Deploying models is the key to making them useful.
It’s not only if you’re building product, in which case deployment is a necessity — it also applies if you’re generating reports for management. Ten years ago it was unthinkable that execs wouldn’t question assumptions and plug their own numbers into an Excel sheet to see what changed. Today, a PDF of impenetrable matplotlib figures might impress junior VPs, but could well fuel ML skepticism in the eyes of experienced C-suite execs.
Don’t help bring about the end of the AI hype cycle!
And so, deployment of ML models became the hot topic, simply because there aren’t that many people who know how to do it; seeing that you need both data science and engineering skills. As I recently discovered, there are two really divergent ways to deploy models: the traditional way, and a more recent option that, I will be honest, blew my mind.
In this article, I’ll provide you with a straightforward yet best-practices template for both kinds of deployment. As always, for the kinaesthetic learner, feel free to skip straight to the code here, which I actually deployed here if you want to test it out. I know not everybody likes to jump around when reading; it looks like this:
If you come from an analyst background, you may not grok web-app architecture, so let me illustrate that first. Apologies if this is oversimplification and man-splaining! But I have seen enough “ML model deployments” that are really just XGBoost wrapped in Flask, that I know it’s a real problem.
The user (on the left here) is using a browser that runs only Javascript, HTML, and CSS. That’s the frontend. It can make calls to a backend server to get results, which it then maybe processes and displays. The backend server should respond ASAP to the frontend’s requests; but the backend may need to talk to databases, third party APIs, and microservices. The backend may also produce slow jobs — such as ML jobs — at the request of the user, which it should put into a queue. (Bear in mind that usually that the user usually has to authenticate itself somehow).
Now, let’s talk distributed web app architecture.
In general, we want to run as many backend instances as possible, for scalability. That’s why there are bubbles coming out of ‘server’ in the diagram above; they represent ‘more of these’. So, each instance has to remain stateless: finish handling the HTTP request and exit. Don’t keep anything in memory between requests, because a client’s first request might go to one server, and a subsequent request to another.
It’s bad if we have a long running endpoint: it would tie up one of our servers (say... doing some ML task), leaving it unable to handle other users’ requests. We need to keep the web server responsive and have it hand off long running tasks, with some kind of shared persistence so that when the user checks progress or requests the result, any server can report. Also, jobs, and parts-of-jobs, should be able to be done in parallel by as many workers as there are resources for.
The answer is a first-in, first-out (FIFO) queue. The backend simply enqueues jobs. Workers pick and process jobs out of the queue, performing training or inference, and storing models or predictions to the database when done.
With the library MLQ, the following is literally all you need for a backend web server — an endpoint to enqueue a job, an endpoint to check the progress of a job, and an endpoint to serve up a job’s result if the job has finished.
The architecture for truly deploying an ML model is this:
Backend server receives a request from user’s web browser. It’s wrapped up in JSON but semantically would be something like: “Tomorrow is Wednesday and we sold 10 units today. How many customer support calls should we expect tomorrow?”Backend pushes the job {Wednesday, 10} into a queue (some place decoupled from the backend itself, such as Redis in the case of MLQ). The queue replies with “Thanks, let’s refer to that as Job ID 562”.Backend replies to the user: “I’ll do that calculation. It has ID 562. Please wait”. Backend is then free to serve other users.The user’s web browser starts displaying a ‘please wait’ spinner.Workers — at least, ones that are not currently processing another job — are constantly polling the queue for jobs. Probably, the workers exist on another server/computer, but they can also be different threads/processes on the same computer. Workers might have GPUs, whereas the backend server probably does not need to.Eventually, a worker will pick up the job, removing it from the queue, and process it (e.g. run {Wednesday, 10} through some XGBoost model). It’ll save the prediction to a database. Imagine this step takes 5 minutes.Meanwhile, the user’s web browser is polling the backend every 30 seconds to ask if job 562 is done yet. The backend checks if the database has a result stored at id=562 and replies accordingly. Any of our multiple horizontal backends is able to serve the user’s request. You might imagine that the shared database is a single point of failure, and you’d be right! But separately, we provisioned replicas and some failover mechanism, maybe sharding/load balancing, so it’s all good.After five minutes plus a bit, the user polls for a result, and we are able to serve it up.
Backend server receives a request from user’s web browser. It’s wrapped up in JSON but semantically would be something like: “Tomorrow is Wednesday and we sold 10 units today. How many customer support calls should we expect tomorrow?”
Backend pushes the job {Wednesday, 10} into a queue (some place decoupled from the backend itself, such as Redis in the case of MLQ). The queue replies with “Thanks, let’s refer to that as Job ID 562”.
Backend replies to the user: “I’ll do that calculation. It has ID 562. Please wait”. Backend is then free to serve other users.
The user’s web browser starts displaying a ‘please wait’ spinner.
Workers — at least, ones that are not currently processing another job — are constantly polling the queue for jobs. Probably, the workers exist on another server/computer, but they can also be different threads/processes on the same computer. Workers might have GPUs, whereas the backend server probably does not need to.
Eventually, a worker will pick up the job, removing it from the queue, and process it (e.g. run {Wednesday, 10} through some XGBoost model). It’ll save the prediction to a database. Imagine this step takes 5 minutes.
Meanwhile, the user’s web browser is polling the backend every 30 seconds to ask if job 562 is done yet. The backend checks if the database has a result stored at id=562 and replies accordingly. Any of our multiple horizontal backends is able to serve the user’s request. You might imagine that the shared database is a single point of failure, and you’d be right! But separately, we provisioned replicas and some failover mechanism, maybe sharding/load balancing, so it’s all good.
After five minutes plus a bit, the user polls for a result, and we are able to serve it up.
There is a little bit more involved, mostly to handle resiliency and persistency (what if a worker goes offline midway through a job? What if the user’s input was garbage and caused the job to fail?) But that’s the basics. Here’s a very simple worker template for MLQ. It just waits until it receives a job, then runs a function on the job parameters and stores the result. You can run as many of these things in parallel as you want, on the same server or distributed servers. If you look in the repo, you’ll find the complete code for doing this with the Nietzche/Tensorflow RNN model.
There are several good queueing frameworks available, or things that make suitable queues, including Celery, Dask, ZeroMQ, native Redis, and a library I recently made to be an easy-to-use version of all this for deploying side projects without complexity: MLQ. Kafka is also a thing but regular readers will know I’m not a fan of over-architected, Java-based projects. MLQ is immature; I’m not trying to sell it here. Use Celery instead for serious projects.
This week, I spent some time with NVIDIA and asked about their canonical solution for job queueing (specifically, in my case, so that I can make a GPU farm available to everyone at work with a Jupyter notebook, without them all trying to submit jobs at the same time). There isn’t one, yet, but I was assured they’re working on it. Until then, hand-rolling a solution with a queuing system is the only way.
(Also possibly of interest from that meeting: everyone agreed that MXNet is a really good framework, perhaps the best — but sadly may be on its way out).
You might be wondering, how does ML queuing work with real time applications? The answer is: the same way, but it’s not ideal due to latency (for example, industrial IoT). Queue entrypoints can be distributed, so the real tricks are in how the database handles that. Additionally, general opinion is that people won’t accept private data being sent to the backend, another reason why “ML at the edge” is a hot topic. If all the data needed for inference is available in one place, let’s do the inference there. And so, without further adieu:
And so, enter the poor maligned front end engineer, whom everyone assumed thought linear algebra meant doing calculations one after the other, but who is repeatedly the most in-demand person on your team. Who, it turns out, probably wasn’t a nitwit after all, but maybe just biding their time until Javascript’s AI tools caught up to Python’s. And recently, they did.
The nutshell is, you can use Tensorflow from Javascript now. Initially I dismissed Google’s announcement of this thinking it probably meant running inference with hobbled, crippled models which have to fit into some pattern (a single convolutional layer with a maximum 12 filters, FP-8, etc). And only inference! Definitely not training. How could that even be possible from Javascript.
I was very wrong!
I don’t want to focus on training a model in Javascript in this article — that’s super cool, but not always super practical — but rather, provide an alternative deployment pattern for a trained model. Bear in mind that your trained model will be available to the world. Anyone can copy it and see what the layers look like, and steal all the parameters. I guess I’d say that’s inevitable and your model probably is less special than you imagine: any competitive advantage is in the data and speed that you can deploy model revisions. And of course, how great of a product you build atop the model. Anyway, be wary of that.
TensorflowJS can execute, in a user’s web browser, any Keras model. And, via Web GL, they are hardware accelerated! I don’t have hard numbers but anecdotally, it’s worked well for me. Definitely not as fast as Python, but I am sure that over time JS will catch up.
For this article, I copied the code from the official Tensorflow Keras text generation LSTM example and ran it to build a model. My complete Jupyter notebook is here.
Then, export the model to TFJS. You’ll probably need to pip install tensorflowjs. Then:
import tensorflowjs as tfjstfjs.converters.save_keras_model(model, '.')
In the directory you’ll now find model.json and group1-shard1of1.
All the Tensorflow JS examples right now use Yarn, which I understand is a bit outdated (back when I was doing more visualization work, yarn became the new hotness versus npm's old dog; now it’s gone back the other way). Let’s for now put aside front-end’s capriciousness.
Anyway, hopefully you have yarn and a working node installation (version 9 at least). For a minimal example of a website that serves a frontend model, you can clone my repo.
The actual Javascript code is not that interesting. There’s a bit of boilerplate around creating a tensor from a buffer, but all you need to do to actually use the model:
model = await tf.loadModel('https://mydomain.com/model.json');output = model.predict(input);
The complete, end-to-end, frontend (and backend) deployment is example is in my repo here.
Viola! Hardware accelerated Keras models where you don’t even need a backend.
The biggest drawback I can think of, other than your network architecture being available for all to see, is that in many practical applications not all of the data is available at the front end. Where I work, when a user enters a query, we fetch a massive amount of data from Elasticsearch, and run a model (actually several) on that data. Sending so much data to the frontend is not feasible.
Even if it was, you might want to charge per-prediction, which would not be possible once you have entered Javascript land.
It’s simple:
Use a queue
Don’t tie up your backend webserver; separate any ML processes from the act of serving up assets and endpoints
Ensure everything is stateless and able to operate in parallel
Consider frontend deployment
I hope you enjoyed and gained something from this article! Please like and follow if you did, and/or provide feedback here or at @tomgrek. I build AI in SF, and always enjoy talking to fellow nerds/AI enthusiasts, so feel free to get in touch via whatever means.