A Beginner's Guide to Connect Celery With MongoDB
By: Vikas Gautam
Tags: Programming, Database, Celery, Python, Mongodb

Project Directory Structure

---Blogs

|---Blogs

|---__init__.py

|---settings.py

|---urls.py

|---celery.py

|---UserRegistration

|---tasks.py

|---utils.py

|---BlogTasks

|---tasks.py

|---urls.py

|---views.py

manage.py

Let’s Connect These In three Steps

1. Connect Django and MongoDB

In your settings.py file. DBName mentioned here will be created, if not already present.

import mongoengine mongoengine.connect(db='YourDBName', host='127.0.0.1', port='27017')

2. Connect Celery And Redis

In your settings.py file.

# CELERY SETTINGS

CELERY_BROKER_URL = 'redis://localhost:6379/0'

CELERY_ACCEPT_CONTENT = ['json']

CELERY_TASK_SERIALIZER = 'json'

CELERY_RESULT_SERIALIZER = 'json'

3. Connect Celery and MongoDB

In your settings.py file.

# CELERY MONGO SETTINGS

CELERY_RESULT_BACKEND = "mongodb"

CELERY_MONGODB_BACKEND_SETTINGS = {

"host": "127.0.0.1",

"port": 27017,

"database": "jobs",

"taskmeta_collection": "stock_taskmeta_collection",

}

A collection named jobs will be created in your DB, this collection will hold metadata generated by the execution of tasks.

celery.py

Now we need to insert a task in our MongoDB. That task will be picked by the celery beat and executed by the worker.

To insert a task we just need to run a method that will have all tasks. Check the example below.

BlogTasks/tasks.py

Run create_tasks and this will insert a record in the schedule collection. Here you can see we have created a task named user-account-creation-task. A method executed by user-account-creation-task is present in the location UserRegistration.tasks and the method is user_account_creation_task .

The frequency of user-account-creation-task is eight hours. But you can also do something like schedule[crontab][‘minute’] = ‘*/5’ and schedule[crontab][‘hour’] = ‘*’ this means that every five minutes task will be executed. But if you didn’t change it to per eight hours then your task will run every eight hours five mins.

Also, note that we are using celerybeat-mongo GitHub project that we have already installed to create a task, validate data (using PeriodicTaskSerializer ), and saving in MongoDB.

UserRegistration/tasks.py

from celery import task

from UserRegistration.utils import account_creation

@task

def account_creation():

account_creation()

UserRegistration/utils.py

def account_creation():

# Apply your logic here

print('Running account creation task!')

Now we have all the required libraries and code in files we can run celery worker and beat and check the result.

Run mongod service

Linux Env

service mongod start

Windows CMD

add mongod location in the environment variables

C:\Program Files\MongoDB\Server\3.4\bin\

Start MongoDB service using cmd mongod

Redis service

Linux Env

service redis start

Windows CMD

add redis in environment variables

C:\Program Files\Redis\

start redis service using redis-cli

Celery Worker

First, change your directory to where the manage.py file is present

celery -A Blogs worker -l info

If you are getting any errors then install eventlet and run

celery -A Blogs worker -l info -P eventlet

Celery Beat

First, change your directory to where the manage.py file is present:

celery -A Blogs beat -S celerybeatmongo.schedulers.MongoScheduler -l info

Here we’re using a database scheduler which is provided by celerybeat-mongo . Its role is to pick tasks from the database and send it to the worker.

If everything is done correctly then in your worker cmd you will see a print statement result Running account creation task!

Why Should We Use the Database Scheduler?

At CloudGain we prefer to use Database Scheduler because all of the schedules are present in MongoDB. That means we can easily manipulate frequencies of tasks and run them without the need for deployment. This makes testing tasks very easy.