Dynamic Task Scheduling With Django-celery-beat
By: Meriç Melike Yılmaz
Tags: Backend, Task Scheduling, Celery, Django

When there is a need to run asynchronous or recurring tasks in Django applications, celery is the go-to project for implementation. And celery docs and the examples out there are quite enough to get started. However, there is a special use case we will cover here: dynamically starting, pausing, or stopping periodic tasks depending on the state of our models or the user input.

This guide assumes that you already know how to set up a Django project, install requirements in a virtual environment, run the server, and play around with models, views, and serializers. We will cover setting up celery , django-celery-beat , and the necessary models and signals to play with periodic tasks. If you want to skip ahead and dive into the action directly, check out the example Github repository¹.

Celery — Distributed Task Queue

Celery is a task queue with focus on real-time processing, while also supporting task scheduling.²

There are two main usages of celery in a regular Django application. 1) Queuing an operation to run asynchronously in a celery worker, and 2) Scheduling a task to run either once or regularly in the background of the application. celery-beat acts as the scheduler part of celery whereas the worker executes the tasks that are either instructed from within the application or by celery-beat .

The installation steps for celery in a Django application is explained in celery docs here (after pip install celery 🙃).

We will set up Redis to act as the message broker between celery and our app. To accomplish that we’ll ensure we have a Redis server running locally and add the following settings in our application:

CELERY_BROKER_URL = 'redis://localhost:6379' CELERY_RESULT_BACKEND = 'redis://localhost:6379'

Depending on the needs of your project you can use other brokers like RabbitMQ, but we really like Redis.

django-celery-beat

django-celery-beat extension stores the schedule in the Django database, and presents a convenient admin interface to manage periodic tasks at runtime.³

Before we move onto the ‘dynamic’ part of the story, we will set up django-celery-beat which will allow us to add and remove periodic tasks in our application. Once we confirm that we can schedule jobs and ensure that they are picked up by our celery workers, we’ll move on to binding those schedules to our models and starting and stopping tasks on the fly.

Let’s install the extension:

pip install django-celery-beat

We will also add django-celery-beat to the list of installed apps in our Django app settings. That’s because this library introduces new models where the schedules and tasks will be kept; and our application needs to know about that to be able to migrate our database accordingly and create the necessary tables for the extension to work.

INSTALLED_APPS = (

...,

'django_celery_beat',

)

Now we will run the migrations introduced by the extension to create the related tables;

python manage.py migrate

One last step and we are good to go! We will add a new setting to let celery know to use our new scheduler when running celery beat. So add this to your settings.py :

CELERY_BEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'

Confirming our Setup

If everything went fine, we should now have the following list of models to play with inside Django admin:

I went into Periodic tasks and have created a new periodic task with the name Hello World to run every 15 seconds. As the registered task, I chose django_dynamic_periodic_tasks.celery.hello_world which I defined in my celery.py like below as a task to test my changes. I chose now as the start date and time so it starts to run immediately.

@app.task(bind=True) def hello_world(self): print('Hello world!')

So what is going on here? As mentioned before, celery-beat is the part of celery that is in part of scheduling tasks, and in this case, it uses the models in our DB to create the tasks and send them off to the celery worker because of the beat_scheduler we declared in our settings. Now by looking at the periodic task instance in the DB, celery-beat will know to run the task when its time comes.

We will now run our celery beat and worker processes to get the ball rolling.

(env) ➜ django-dynamic-periodic-tasks celery -A django_dynamic_periodic_tasks beat -l INFO

celery beat v5.0.2 (singularity) is starting.

...

[2020-12-03 15:05:05,384: INFO/MainProcess] beat: Starting...

[2020-12-03 15:05:20,405: INFO/MainProcess] Scheduler: Sending due task Hello World (django_dynamic_periodic_tasks.celery.hello_world)

That should be the command and the output for the beat ; now let’s see the worker part (restarted a minute later for a clean output, so don’t mind the timestamps):

(env) ➜ django-dynamic-periodic-tasks celery -A django_dynamic_periodic_tasks worker -l INFO

...

[2020-12-03 15:06:16,670: INFO/MainProcess] Connected to redis://localhost:6379//

[2020-12-03 15:06:16,680: INFO/MainProcess] mingle: searching for neighbors

[2020-12-03 15:06:17,707: INFO/MainProcess] mingle: all alone

[2020-12-03 15:06:17,732: INFO/MainProcess] celery@Merics-MacBook-Pro.local ready.

[2020-12-03 15:06:20,404: INFO/MainProcess] Received task: django_dynamic_periodic_tasks.celery.hello_world[54c86211-e474-4c8f-834c-59f0a81ec503]

[2020-12-03 15:06:20,406: WARNING/ForkPoolWorker-2] Hello world!

[2020-12-03 15:06:20,414: INFO/ForkPoolWorker-2] Task django_dynamic_periodic_tasks.celery.hello_world[54c86211-e474-4c8f-834c-59f0a81ec503] succeeded in 0.007474240999999715s: None

Our task Hello World now runs every 15 seconds. We are good!

Now django-celery-beat is already perfect when you want to manage your intervals and tasks through Django admin. So if you don’t need to integrate your scheduling in your existing structure, you don’t need to move further than this section. But as discussed, we’ll want to dynamically add tasks to our application when the user creates a new model instance. So let’s actually do that thing where our tasks will depend on the model instances of our application 💃

Dynamic Periodic Tasks

Here is what we want to do:

We will have a model called Setup which the user will be able to create, disable, or delete whenever they like. When creating a Setup , the user will select an interval that we will define in an enum, i.e. a choice between 1 minute , 5 minutes , 15 minutes , 1 hour , and so on. We will have a computation-heavy task computation_heavy_task that we want to regularly run for each of our setups with the given intervals. For each Setup that is active, celery will execute the computation_heavy_task every 1 minute, 5 minutes, 15 minutes, etc, depending on what interval has been chosen for that Setup.

You could also just create, enable/disable PeriodicTask models in your interface, but our use-case requires us to run several of these tasks with different configurations and variables depending on the setup, so we’ll couple it with a Setup model.

Setup model

Let’s add the basics for the Setup model:

# setups/models.py import json from django.db import models from django.utils import timezone from django_enum_choices.fields import EnumChoiceField from django_celery_beat.models import IntervalSchedule, PeriodicTask from .enums import TimeInterval, SetupStatus

class Setup(models.Model): class Meta: verbose_name = 'Setup' verbose_name_plural = 'Setups' title = models.CharField(max_length=70, blank=False) status = EnumChoiceField(SetupStatus, default=SetupStatus.active) created_at = models.DateTimeField(auto_now_add=True) time_interval = EnumChoiceField( TimeInterval, default=TimeInterval.five_mins) task = models.OneToOneField( PeriodicTask, on_delete=models.CASCADE, null=True, blank=True )

And the enums:

# setups/enums.py from enum import Enum class TimeInterval(Enum): one_min = '1 min' five_mins = '5 mins' one_hour = '1 hour' class SetupStatus(Enum): active = 'Active' disabled = 'Disabled'

Listening for Setup save

So our setup model will have a title , a status with options Active and Disabled , a created_at timestamp, a time_interval enum, and a task of type PeriodicTask provided by django-celery-beat . The task field is optional because we will use Django’s model views later and we don’t want to bother creating a task from within the view. Instead, we’ll create a PeriodicTask in the signal that is triggered when a Setup model instance is created. If you are not familiar with signals in Django, check the documentation here.

Mind that we also need to disable the task when a Setup instance status is set to Disabled , and vice versa, so in the signal, we’ll watch out for the status as well.

Assuming that we have a setup_task model function for Setup , our signal can look like this:

@receiver(post_save, sender=Setup) def create_or_update_periodic_task(sender, instance, created, **kwargs): if created: instance.setup_task() else: if instance.task is not None: instance.task.enabled = instance.status == SetupStatus.active instance.task.save()

This is maybe the most important part of our flow 🙌 Here we say:

When a Setup is created, set up its task! When a Setup is updated, if it has a task associated with it, set its enabled status to Setup 's active status.

We choose to write a signal here instead of manipulating the setup method’s save ; because we will save Setup 's id as an argument to the task to be able to access our setup from within the task.

Creating the periodic task

Let’s now add our setup_task method.

When you look at django-celery-beat 's docs you’ll see that a PeriodicTask depends on a schedule model to have been created. We could dynamically generate that as well, but we chose an enum over a DurationField etc., and we want to limit the number of scheduling options the user will have. So at this point, we’ll go to Django Admin in our application and manually create Intervals for 1 minute, 5 minutes, and 1 hour. It’s quite simple to do with the models and admin options provided by the extension. We’ll use these in our implementation to determine the interval property of the PeriodicTask that we will create. Now we can actually add the methods to create the task for each setup! Just add the following to your Setup model.

def setup_task(self): self.task = PeriodicTask.objects.create( name=self.title, task='computation_heavy_task', interval=self.interval_schedule, args=json.dumps([self.id]), start_time=timezone.now() ) self.save() @property def interval_schedule(self): if self.time_interval == TimeInterval.one_min: return IntervalSchedule.objects.get(every=1, period='minutes') if self.time_interval == TimeInterval.five_mins: return IntervalSchedule.objects.get(every=5, period='minutes') if self.time_interval == TimeInterval.one_hour: return IntervalSchedule.objects.get(every=1, period='hours') raise NotImplementedError

Let’s look at what setup_task does in more detail. It creates a PeriodicTask model with a name of the Setup 's title , to run our computation_heavy_task , starting now, every interval . And as an argument, the task will receive the id of our Setup , which means it will have full access to the Setup and anything else we might require from the database.

Before we see what our task should actually look like, there is one more step we should complete here. Notice that we declared task field of the Setup model as a OneToOneField with on_delete=models.CASCADE . This means when we would manually delete a PeriodicTask from the app, its related Setup would also be deleted. But Django doesn’t automatically support the case the other way around; we will proactively have to delete the PeriodicTask when a Setup is deleted. We can accomplish that by overriding Setup 's delete like so:

def delete(self, *args, **kwargs): if self.task is not None: self.task.delete() return super(self.__class__, self).delete(*args, **kwargs)

The celery task

The only thing we haven’t looked at so far is the task itself. Since the possibilities are endless here, we’ll settle with a simple print statement here.

from celery import shared_task from .models import Setup @shared_task(name="computation_heavy_task") def computation_heavy_task(setup_id): setup = Setup.objects.get(id=setup_id) # Do heavy computation with variables in setup model here. print('''Running task for setup {setup_title}.'''.format( setup_title=setup.title))

Note that if you have a celery worker running at this point, you will have to restart the worker for the new task to be registered.

Checking the results

Everything should now be in order! Make sure your celery worker and celery-beat are running, your migrations are executed, you have a superuser, and go to Django admin. When you create a Setup instance there; you’ll see in the celery logs (when the time comes 😉) that the task is running periodically. When you disable a Setup , the task will stop running, which then you can enable back. More importantly, it is super extendable so you can add whatever you like to your Setup model, as well as to the celery task you want to run. Make sure to examine the repo for django-celery-beat as well, since you might want to make use of more of their features like CrontabSchedule , or more fields for the task that haven’t been mentioned here.

Tl;dr

We created a Setup model in a simple Django application, and with that, we made it possible to dynamically add a periodic celery task every time a Setup is created. We also have the option to disable the tasks temporarily by setting the setup to disabled status; or we can delete it altogether. Our task has access to our setup 's id so we can customize the task to use different variables or configurations for every setup that we have.

We only worked on the backend part of the things here and didn’t delve into the templates or forms; so in order to test everything we used Django admin and the output from our celery worker.

If you have any questions or any ideas to improve this further, drop me a message or leave a response below!

[1]: https://github.com/melikesofta/django-dynamic-periodic-tasks

[2]: https://docs.celeryproject.org/en/stable/

[3]: https://docs.celeryproject.org/en/stable/userguide/periodic-tasks.html#using-custom-scheduler-classes