Job Queuing 101: Start using Bull in your Node.js Project (Part I)
By: Rodrigo Luque
Tags: Job Queue, Koa, Nodejs, Redis, Queue

Let’s begin

Installation

We will start by creating a Koa server with the basics. So we will execute the following commands on the terminal:

$ npm init -y // To initialize a Node project $ npm install koa // Our application is ready to use Koa $ npm install @koa/router // Router for our koa application $ npm install koa-bodyparser // Parses the body in request $ npm install nodemon --save-dev // For running our server

Once the package.json file is generated. we will need to do 2 things. Firstly, we will obviously need to install two packages: bull and bull-board. The last one is a third-party library developed on top of bull to help you visualize your queues and their jobs with a UI.

$ npm install bull $ npm install @bull-board/koa // UI library

As I mentioned before, Bull works with Redis as we need to store/cache the job-description of each of the jobs, so the next thing we will have to do is to install Redis on our machine. This process can be done in 2 ways; either install Redis on your computer, or use Redis with Docker.

I am going to do it using the second way, which can be done pulling the image from here. In case you want to do it using the first way, here I leave you a link to install it on a Mac.

We need the Redis server to be up and running when we run our project.

First steps

Let’s start by creating a folder named src with our entry file for the application which will be named index.js , and setting up our Koa server.

src/index.js

We have just created a simple Koa server with a router that has an endpoint that indicates if our server is running.

Creating our First Queue

We want to create a queue that handles the orders that we are going to be receiving, being our producer. So let’s start by creating a folder named queues and a file named order-queue.js :

src/queues/order-queue.js

With the code above, all we did is to create our orders queue, passing our Redis url, which in this case will be the local one (redis://localhost:6379) as Redis by default listens in port 6379.

Then, we created the createNewOrder method that will receive an order and insert it into our queue. For now, we will leave the options empty but in the future we will see what can we put here.

Creating our First Consumer

We will create a new file named orders-queue-consumer.js where we will indicate the process function. We will export it as we will use it in our queue declared before.

src/queues/orders-queue-consumer.js

The process function will be called every time the worker is idling and there are jobs to process in the queue.

Now, we just need to add the process function created as the callback for the queue process function. By just adding this:

src/queues/order-queue.js

We just added the function that we exported before and indicated it in ordersQueue.process(ordersProcess) .

Time to Test our First Order

All we want to do now is to test if everything is working fine. To do that, we are going to create and endpoint that will simulate a new order received in our online store, so let’s create it real quick in our index.js file under the route /order as a POST method.

src/index.js

Now, in order to see it all working, we just go to Insomnia (or similar), and make a request to this endpoint. What it should happen is that, besides receiving the response with a 200 status of course, we should check in our Redis local database that the order has been queued. So let’s start our server and make the request.

POST request done in Insomnia

In order to check our Redis storage, I personally use an extension available for VSCode named Redis Client. After doing the request, we get that a job was enqueued with its properties as seen below.

Job enqueued in our orders queue

And we will see that the process log has been invoked and therefore, the order was logged as shown below: