Use Redis Queue for Asynchronous Tasks in a Flask App | by Edward Krueger | Towards Data Science

By: Content by Edward Krueger, Josh Farmer and Douglas Franklin.
When building an application that performs time-consuming, complex, or resource-intensive tasks, it can be frustrating to wait for these to complete within the front end application. Additionally, complex tasks in the front end can time-out. Redis Queue fixes this by pushing more sophisticated tasks to a worker for processing.
Using Redis with Redis Queue allows you to enter those complex tasks into a queue, so the Redis worker executes these tasks outside of your application’s HTTP server.
In this article, we will build an app that enqueues jobs with Redis queue, performs a function on those jobs and returns the result of the function.
Here is the link to the Github Repository with our code for this project.
Redis is an open-source, in-memory database, cache, and message broker. Messages handled by Redis are essentially JSONs. Redis is ideal for quickly and easily working with specific data types in a temporary database, and provides very rapid access and delivery of queries.
For us, Redis offers two benefits. First, it pushes complex tasks to another space to be processed. Second, it is easier for the developer to handle complex actions by splitting the task into separate functions, the main application and the queue.
For this application, we’ll be using Redis to hold our queue of JSON messages. Redis can be a stand-alone database, accessible by many computers or systems. In our example, we will be using it as a local memory store to support our application.
Redis Queue is a python library for queueing jobs for background processing. Since many hosting services will time out on long HTTP requests, it is best to design APIs to close requests as quickly as possible. Redis Queue allows us to do this by pushing tasks to a queue and then to a worker for processing.
Using Redis in conjunction with Redis Queue allows you to request input from the user, return a validation response to the user, and queue up processes in the background. All without the front end user having to wait for those processes to complete. Processes could be anything from Machine Learning models, to duplicating an image to complex simulations.
Anything that takes longer to complete than you would like to have taken place on the front end of your application belongs in the Redis Queue.
We will cover using Docker to run the Redis database and initialize the Redis Queue worker. The worker will allow us to process the jobs in our application’s queue.
We’ll go over each of these files in more detail in the following sections.
We’ll discuss the following constituent files of our app, main.py, functions.py, and redis_resc.py.
There are two ways to initiate the docker container. We will be covering the standard process one step at a time in this article. However, the docker container can be created using docker-compose, which we will cover in a separate article.
Docker is a containerization service. This means Docker runs code within a container that has within it the dependencies required for the application to run reliably from one system to another. It ensures our applications run consistently, from development to testing to production.
Follow this link to the Docker documentation for a comprehensive installation guide to help you select the correct version of Docker for your operating system.
Once you have Docker installed on your system, you will need to install Redis.
For Mac users, the home-brew command brew install redis will work. Windows requires a download from Github. With Linux, you can download the latest tarball directly from Redis.
To run our application, we must understand the instructions on the Readme file.
We will be using Pipenv to create a virtual environment and install the appropriate packages. Install the packages for this program with the command pipenv install -- dev. Adding the --dev flag installs development packages and production requirements. Before moving to the next step, activate the environment with the command pipenv shell.
We will have to start three services to have the app function correctly: Redis Database, a Redis Queue worker, and the Flask application. Two of these services will begin within the same terminal as the one we used to install our virtual environment and enter the shell.
First, run the command:
docker pull redis
This will pull the Redis image from Docker Hub.
Then, run the command:
docker run -d -p 6379:6379 redis
This command serves several purposes. First, it initializes the docker container. The -d flag runs the Redis container in the background, freeing up your terminal for the next step. The -p flag publishes your container’s port or ports to the host. Lastly, 6379:6379 binds port 6379 inside the docker container to port 6379 on the localhost. Without binding the port, you will run into issues when the local machine and processes within your docker container attempt to communicate with one another.
The final step before running our application is starting the Redis Queue worker. Since we are running the Redis image in detached mode, we can do this step within the same terminal.
Start the Redis worker with the command rq worker . The rq command is only available to the terminal if the Redis Queue package is installed, we installed the dependencies with Pipenv install, and entered the environment with pipenv shell. Running the rq command will allow Redis Queue to begin listening for jobs to enter into the queue and start processing these jobs.
We are now ready to initialize the flask app. On a development server, this is done with the command:
export FLASK_APP=app.main:app && flask run — reload
The command gunicorn app.main:app will run this on a production server. Gunicorn requires a Unix platform to run. Therefore, this command will work if you are using macOS or a Linux based system, but not for windows.
Now that the services are running, let’s move to the app files.
This file sets up the Redis connection and the Redis Queue.
The variable redis_conn defines the connection parameters for this connection, and redis_queue uses the Queue method. The Queue method initiates the queue, and any name can be given. Common naming patters are ‘low,’ ‘medium’ and ‘high.’ By providing the Queue method no args, we are instructing it to use the default queue. Aside from the name, we are simply passing our method the connection string to the Redis store.
Functions.py is where we define the functions to use in the Redis queue.
What some_long_function does is up to you. It can be anything you want with done with the data passed to the worker. This could be anything from machine learning, to image manipulation, to using the information to process queries in a connected database. Anything you want the backend to do so the front end doesn't wait would be placed within this function.
For our example, this function serves a couple of purposes. First, the job variable is created to retrieve the current job being worked. We use time.sleep(10) to demonstrate the different job status codes as the job is being processed. We will go into more detail on the job status codes later in the article. Finally, we return a JSON that contains information about the job and the results of processing the job.
Main.py is the primary flask app that drives the application. More details on setting up a flask app are outlined in this medium article.
towardsdatascience.com
The first route function, resource_not_found , is designed to handle any 404 errors that result from incorrect requests to the application. The route function defined as home is the default route that runs when you navigate to the home route. This route runs when you perform a GET request to http://127.0.0.1:8000 or the localhost URL. This route designed to let you know the flask app is running.
We can test it with Postman as seen below.
The remaining routes, specific for the Redis Queue, will be covered in more detail below.
The /enqueue route creates the task and enters the task into the queue. Redis Queue accepts things such as key-value pairs so that you can create a post request with a dictionary, for example:
{“hello”: “world”}
Submitting a JSON post request to the /enqueue route will tell the application to enqueue the JSON to the function some_long_function within the function.py file for processing.
The /enqueue route returns the job_id of the task created once it has done so. Take note of the job_id, as we will use it as part of the URL string for the application’s remaining routes.
This route takes the job_id and checks its status in the Redis Queue. The full URL of this route is http://127.0.0.1:8000?job_id=JobID. Where JobID is the job_id that was generated when the /enqueue route created the job.
The /check_status route will return a JSON that contains the job_id passed in the URL and the status of the job using the get_status method of rq.job. Remember within the some_long_function, the time.sleep at the start of the function. If you hit this route with a get request before the ten-second timer, the status will show ‘Queue.’ After some_long_function completes, the status will return as ‘finished.’
Once a job is shown as ‘finished’ by the /check_status route, get_result will return the results of the processed job. The results are determined by what you have set up within some_long_function. This URL follows the same structure as /check_status so it is:
http://127.0.0.1:8000?get_result=JobID
Our some_long_function returns data about the job, as seen in the image above.
In this article, we learned a little about building a queue of tasks with Redis Queue, and how to utilize a Redis database within a docker container. We covered how to build the application that makes it simple to enqueue time-consuming tasks without tying up or hindering our front end performance.
github.com
Here is the link to the Github Repository with our code for this project. Test it out for yourself and try queueing a few JSON tasks and viewing the results!