5 Ways to Schedule Jobs in Python
By: Pratik Choudhari
Tags: Programming, Coding, Python, Software Development, Redis

Simple loops

This is a no-brainer. Using infinitely running while loops to periodically call a function can be used to schedule a job, not the best way but hey it works. Time delay can be given using the sleep function of the in-built time module. This is not exactly how most jobs are scheduled because first, it looks ugly and second, it’s less readable compared to other methods.

When it comes to scheduling routines like at 9:00 am every day or 7:45 pm every Wednesday, things get tricky.

My first reaction? Nope thank you! A problem with this approach is that the logic here is blocking i.e., once python discovers this piece of code in a project, it will get stuck in while 1 loop hence, blocking the execution of other code.

Simple loops but threaded

Threading is a concept in computer science where threads, small programs with their own instructions, are executed by a process and are managed independently. This can resolve the blocking nature of our 1st approach, let’s see how.

After a thread is started, its underlying logic cannot be modified by the main thread, therefore, we may need to add resources through which the program can check for specific scenarios and execute logic based on them.

Schedule Library

Earlier, I said scheduling using while loop looks ugly, schedule library takes care of that.

As you can see, Multiple schedules can be created effortlessly. I especially like the way of creating jobs, the method chaining, on the other hand, this snippet has a while loop which means the code is blocking and you already know what can help us here.

Photo by John Anvik on Unsplash

Python Crontab

The crontab utility in Linux is an easy-to-use and widely accepted scheduling solution. A Python library, python-crontab, provides an API to use the CLI tool from Python. In crontab, a schedule is described using the unix-cron string format ( * * * * * ), which is a set of five values in a line, indicating when the job should be executed. python-crontab transforms writing the crontab schedule in a file to a programmatic approach.

Image by Linux hint

python-crontab doesn’t auto-save the schedules, the write() method needs to be executed to save the schedule. There are many more features and I urge you to have a look at their documentation.

RQ Scheduler

Some tasks cannot be executed immediately therefore we need to create a task queue and pop tasks according to a Queue system such as LIFO or FIFO. python-rq allows us to do exactly this, using Redis as a broker to queue jobs. The entry for a new job is stored as a hash map with info such as created_at, enqueued_at, origin, data, description .

Queued jobs are executed by a program called worker. Workers too have an entry in the Redis cache and are responsible for dequeuing a job along with updating the job status in Redis. Jobs can be queued as and when required but to schedule them we need rq-scheduler.

An RQ worker must be started separately in a terminal or via python-rq worker utility. As soon as a job is triggered, it can be seen in the worker terminal, separate function callbacks can be used in success and failure scenarios.

Conclusion

There are a few more libraries for scheduling but here, I have discussed the most common ones. An honorable mention would be Celery, added advantage of celery is that the user can choose between multiple brokers. I appreciate you reading till the end. Check out my other articles too. Cheers!

More content at plainenglish.io