Package: django-cq
Django-CQDescriptionAn attempt to implement a distributed task queue for use with Django channels.
Modelled after RQ and Celery, complex task workflows are possible, all leveraging
the Channels machinery.WhyThere are three reasons:Aiming for more fault tolerant tasks. There are many occasions where information
regarding how tests are progressing is needed to be stored persistently. For
important tasks, this should be stored even in the case of a Redis fault, or if
the worker goes down.Prefer to leverage the same machinery as channels.Would like to have a little extra functionality surrounding subtasks that didn't
seem to be available via Celery or RQ.LimitationsThere are two limitations involved:REDIS must be used as the Django cache.asgi_redismust be used as the channel backend.There is work being done to remove these restrictions.InstallationUse pip if you can:pipinstalldjango-cqor live on the edge:pipinstall-ehttps://github.com/furious-luke/django-cq.git#egg=django-cqAdd the package to your settings file:INSTALLED_APPS=['cq',...]And include the routing information in your channel routing list:channel_routing=[include('cq.routing.channel_routing'),...]You'll need to migrate to include the models:./manage.pymigrateYou'll most likely want to create a new channel layer for your CQ
tasks. The default layer has a short time-to-live on the channel messages,
which causes slightly long running tasks to kill off any queued messages.
Update your settings file to include the following:CHANNEL_LAYERS={'default':{...},'long':{'BACKEND':'asgi_redis.RedisChannelLayer','CONFIG':{'hosts':[REDIS_URL],'expiry':1800,'channel_capacity':{'cq-tasks':1000}},'ROUTING':'path.to.your.channels.channel_routing',},}CQ_CHANNEL_LAYER='long'In order to process messages sent on the "cq-tasks" channel a worker
process needs to be launched:./manage.py cq_runworkerTasksBasic task usage is straight forward:@taskdefsend_email(cqt,addr):...return'OK'task=send_emails.delay('dummy@dummy.org')task.wait()print(task.result)# "OK"Here,cqtis the task representation for thesend_emailtask. This
can be used to launch subtasks, chain subsequent tasks, amongst other
things.Tasks may also be run in serial by just calling them:result=send_email('dummy@dummy.org')print(result)# "OK"SubtasksFor more complex workflows, subtasks may be launched from within
parent tasks:@taskdefsend_emails(cqt):...foraddrinemail_addresses:cqt.subtask(send_email,addr)...return'OK'task=send_emails.delay()task.wait()print(task.result)# "OK"The difference between a subtask and another task launched usingdelayfrom
within a task is that the parent task of a subtask will not be marked as complete
until all subtasks are also complete.fromcq.modelsimportTask@taskdefparent(cqt):task_a.delay()# not a subtaskcqt.subtask(task_b)# subtaskparent.delay()parent.status==Task.STATUS_WAITING# True# once task_b completesparent.wait()parent.status==Task.STATUS_COMPLETE# TrueChained TasksTODO@taskdefcalculate_something(cqt):returncalc_a.delay(3).chain(add_a_to_4,(4,))Non-atomic TasksBy default every CQ task is atomic; no changes to the database will persist
unless the task finishes without an exception. If you need to keep changes to
the database, even in the event of an error, then use theatomicflag:@task(atomic=False)defunsafe_task(cqt):passLoggingFor longer running tasks it's useful to be able to access an ongoing log
of the task's progress. CQ tasks have alogmethod to send logging
messages to both the standard Django log streams, and also cache them on
the running task.@taskdeflong_task(cqt):cqt.log('standard old log')cqt.log('debugging log',logging.DEBUG)If the current task is a subtask, the logs will go to the parent.
This way there is a central task (the top-level task) which can be used
to monitor the progress and status of a network of sub and chained tasks.PerformanceDue to the way logs are handled there can be issues with performance
with a lot of frequent log messages. There are two ways to prevent this.Reduce the frequency of logs by settingpublishtoFalseon as many
log calls as you can. This will cache the logs locally and store them
on the nextpublish=Truecall.@taskdeflong_task(cqt):foriiinrange(100):cqt.log('iteration%d'%ii,publish=False)cqt.log('done')# publish=TrueSecondly, reducing the volume of logs may be accomplished by limiting the
number of log lines that are kept. Thelimitoption specifies this. The
following will only keep 10 of the logged iterations:@taskdeflong_task(cqt):foriiinrange(100):cqt.log('iteration%d'%ii,publish=False)cqt.log('done',limit=10)Time-to-liveTODORepeating TasksCQ comes with robust repeating tasks. There are two ways to create
repeating tasks:From the Django admin.Using a data migration.From the admin, click intocqandrepeating tasks. From there you
can create a new repeating task, specifying the background task to call,
and a CRON time for repetition.To create a repeating task from a migration, use the helper functionschedule_task.fromdjango.dbimportmigrationsfromcq.modelsimportschedule_taskfrommyapp.tasksimporta_taskdefadd_repeating(apps,scema_editor):RepeatingTask=apps.get_model('cq.RepeatingTask')schedule_task(RepeatingTask,'* * * * *',a_task)classMigration(migrations.Migration):operations=[migrations.RunPython(add_repeating,reverse_code=migrations.RunPython.noop)]CoalescingPending or queued instances of a coalescing task will prevent other instances of the task from running.