Running Deep Learning Algorithms as a Service | by Nir Orman | Towards Data Science

So, You want to serve Deep Learning Algorithms as a service.
You have a really cool algorithmic library written in Python and TensorFlow/Keras/Some other platform that requires running workloads on a GPU and you want to be able to serve it at scale and have it up and running fast.
Celery is an open-source asynchronous task queue which is based on distributed message passing. After reading all possible blog posts and seeing all the Youtube videos about Celery, I decided it’s the right solution for the task at hand.
Please welcome our main characters of the plot:
The API: Gets a request, creates a Celery async task, and puts it in a queue. (I recommend flask for this task, it’s light but extendable)The message queue: AKA Celery’s Broker. Stores the tasks created by the API in queues. The best practice is to choose RabbitMQ.Workers: a python/celery process which we will run on a GPU and will take tasks from the queues. This is where all the heavy lifting gets done.The result’s backend: Will store the tasks return values. The best practice is to use redis which enables complicated workflows (one task that is dependant on another) without polling.
The API: Gets a request, creates a Celery async task, and puts it in a queue. (I recommend flask for this task, it’s light but extendable)
The message queue: AKA Celery’s Broker. Stores the tasks created by the API in queues. The best practice is to choose RabbitMQ.
Workers: a python/celery process which we will run on a GPU and will take tasks from the queues. This is where all the heavy lifting gets done.
The result’s backend: Will store the tasks return values. The best practice is to use redis which enables complicated workflows (one task that is dependant on another) without polling.
The best practice is to use Celery with RabbitMQ as the broker of the messages and with redis as the result’s backend, in order to use all the unique features that Celery can provide. We know software requirements often change faster than we expect, this should provide us with maximum flexibility so we can use even the most complicated features of Celery. When choosing RabbitMQ and redis, each new task is transformed into a message which Celery then posts to a queue in RabbitMQ, and each return value of a task performed by a worker will automatically be written back to redis (you can easily host RabbitMQ on GCP using “click to deploy”, and redis using AWS Elastic Cache).
Once the message representing the task is in the queue, we need a GPU worker to compute it. The GPU worker will read a message from the queue and perform the task. For example, if it’s a computer vision algorithm, a worker will download the original image from AWS S3, manipulate it, and upload the new image back to S3. The URL of the image will be passed as part of the task.
But wait, there’s a catch.
GPUs, are very expensive machines. One instance of P2.Xlarge in AWS costs over 2,000$ a month (3.06$ an hour while writing these lines) or ~600$ if it’s a spot instance. This obviously means we do NOT want them to be constantly up if there’s no need. They have to be turned on, on-demand, and then turned off. The thing is, Elastic Beanstalk has no feature of auto-scaling according to RabbitMQ queue metrics.
We have to write our own custom Auto Scaler. It’s a big name for a small Python script that runs and polls RabbitMQ for the number of tasks in the queue every 30 seconds. If there’re messages in the queue, it calls the AWS API and makes it start GPU workers accordingly.
Each worker is booted with the docker container of the algorithmic repository (stored in ECR, Elastic Container Registry). Once the container is up and running, it connects to RabbitMQ and redis. It then takes a task from the queue and calculates it. The output is written by the worker to S3. If the task was completed successfully then the return value of the Celery task, is a json containing a URL to the output saved to S3 and its metadata. That return value is automatically saved to redis by Celery and also saved to Postgres DB. If the task failed to finish, an exception is saved to redis.
Check out the diagram below to understand the architecture explained above:
Sounds breezy so far? One of the main challenges using Celery is to configure it the right way.
Here’s a good configuration that’ll save you time and tears when you try to perform Deep Learning tasks at scale. Check it out below and then we’ll dive into every detail of it:
from celery import Celeryfrom api.celery_jobs_app.celery_config import BROKER_URI, BACKEND_URIAPP = Celery(    'celery_app',    broker=BROKER_URI,    backend=BACKEND_URI,    include=['api.celery_jobs_app.tasks'])APP.conf.update({    'imports': (        'api.celery_jobs_app.tasks.tasks'    ),    'task_routes': {        'calculate-image-task': {'queue': 'images-queue'}        }    },    'task_serializer': 'json',    'result_serializer': 'json',    'accept_content': ['json'],    'worker_prefetch_multiplier': 1,    'task_acks_late': True,    'task_track_started': True,    'result_expires': 604800,  # one week    'task_reject_on_worker_lost': True,    'task_queue_max_priority': 10})
Note: The configuration has been simplified in order to make it easier to understand.
Let’s break it down.
The first paragraph in the snippet is just some imports, trivial.
The second paragraph defines the celery app itself, which has a broker and backend (As stated before, the best practice is to use RabbitMQ and redis).
The third paragraph updates the configuration of Celery, This is the interesting part.
The ‘imports’ section says in which of our python packages Celery should look for tasks.
The ‘tasks_routes’ part maps between the task’s name and the queue in which it should be stored. In the code snippet above, all tasks that are of type “calculate-image-task” will be pushed into a queue named “images-queue”. If you do not write which queue your task should be routed to, it will by default go to the default queue named ‘celery’. BTW, you can change the name of the default queue if you want by defining ‘task_default_queue’ property.
FYI: The queue itself is automatically created on RabbitMQ once the first task is routed to it. Cool :)
‘task_serializer’: This is how tasks will be serialized once they are put in the queue and deserialized once they reach the worker. In the image processing case, we do not want the image itself to be serialized and deserialized, The best practice is to store it and only pass its location or URL. We’ll use json as the serializer.
‘result_serializer’: Keep in mind, if you declare the serialization type as json and return a result that is an object or an exception (which is the return type in case there was an exception which wasn’t caught) then your result serialization will throw an exception since any object that’s not a json would throw an exception of failing to serialize. You can always read more about serializers here.
‘accept_content’: A white-list of content-types/serializers to allow.
Tip: It’s not recommended to use ‘pickle’ serializer since it is known to have security issues. Since Celery version 4.0, json is actually the default option of serialization, but “Explicit is better than implicit” (The Zen of Python).
‘worker_prefetch_multiplier’: The default of Celery is that each worker takes 4 tasks and computes all of them before it comes back for the next tasks. Their idea was to optimize network roundtrips. In our case, Deep Learning tasks tend to be long ones (way longer than the network time). This means we do not want a worker to take a bunch of tasks and perform them one after the other. We want each worker to take a single task at a time, and then come back to take the next task when it’s done with the previous one. That way, if one task requires a very long computation, other workers could work on the next tasks simultaneously, since as long as the first worker is not working on them, they are kept in the queue.
‘task_acks_late’: By default, when a worker takes a task, the task will be “acked” just before its execution. In the case of Deep Learning tasks, which take a long time to compute, we would want them to be “acked” only after they are computed. This is especially useful when we use spot instances, which lower our average task price but may also be lost if there’s a shortage of GPU instances and our bidding price wasn’t competitive enough.
‘task_track_started’: Good for tracking that a task has started, cause when your task is long-running, you want to know it’s no longer in the queue (which would be marked as ‘pending’.) I recommend using Flower as the monitor solution for Celery, which allows you to see exactly what’s the status of every task.
‘result_expires’: by default, Celery keeps your results on redis only for 1 day. If you want it to be kept longer, define`result_expires` differently in the configuration file. I would recommend keeping it for at most 1 week and writing the results to a more organized DB that has a schema, such as PostgreSQL.
‘task_reject_on_worker_lost’: We’ll set this to True. When we use spot instances, there’s a chance a worker will be lost when a spot instance is taken away from us. We want the task to be put back into the queue and be computed by another worker. Be careful, if a worker was lost due to hardware errors like ‘out of memory’ etc., then the task will get partially calculated again and again in a loop since the worker will be lost every time it tries to compute it. If you see a task in an endless loop, this is the configuration you should be suspicious about.
‘task_queue_max_priority’: This is where you can make sure that important tasks get done first. You can set a priority for every Celery task (by assigning it with some int representing its priority). If you set this property, you must also set it to your RabbitMQ queue, it does not get set automatically. If a task with priority enters a queue that doesn’t have the priority property, an exception will be thrown and the task will not enter the queue. This property is useful if you have a premium customer whose tasks should be computed first.
If you’re thinking about using this property in order to prioritize fast running tasks over slow ones (such as long GPU computing tasks) then consider adding another group of workers, that are CPU workers and not expensive GPU workers. It would be cheaper and faster.
As you can see in the architecture diagram at the top, You can also have workers running on a totally different cloud.
For example, you could run your workers on Azure AKS which is the Kubernetes of Azure. But that’s a totally different blog post.
Good luck serving your Deep Learning Algorithms with Celery, if you have any questions, feel free to contact me on LinkedIn.