Package: django-pg-queue
django-pg-queuedjango-pg-queue is a task queue system for Django backed by postgres.It was forked from the wonderful and simpler django-postgres-queue (https://github.com/gavinwahl/django-postgres-queue/)
Written by Gavin Wahl.Why postgres?I thought you were never supposed to use an RDBMS as a queue? Well, postgres
has some features that make it not as bad as you might think, it has some
compelling advantages.Transactional behavior and reliability.Adding tasks is atomic with respect to other database work. There is no need
to usetransaction.on_commithooks and there is no risk of a transaction
being committed but the tasks it queued being lost.Processing tasks is atomic with respect to other database work. Database work
done by a task will either be committed, or the task will not be marked as
processed, no exceptions. If the task only does database work, you achieve
true exactly-once message processing.Operational simplicityBy reusing the durable, transactional storage that we’re already using
anyway, there’s no need to configure, monitor, and backup another stateful
service. For small teams and light workloads, this is the right trade-off.Easy introspectionSince tasks are stored in a database table, it’s easy to query and monitor
the state of the queue.SafetyBy using postgres transactions, there is no possibility of jobs being left in
a locked or ambiguous state if a worker dies. Tasks immediately become
available for another worker to pick up. You can evenkill-9a worker
and be sure your database and queue will be left in a consistent state.Priority queuesSince ordering is specified explicitly when selecting the next task to work
on, it’s easy to ensure high-priority tasks are processed first.QueuesSimply implemented by allowing filtering by a queue name in the query.DisadvantagesLower throughput than a dedicated queue server.Harder to scale a relational database than a dedicated queue server.Thundering herd. Postgres will notify all workers who LISTEN for the same name.With at-least-once delivery, a postgres transaction has to be held open for
the duration of the task. For long running tasks, this can cause table bloat
and performance problems.When a task crashes or raises an exception under at-least-once delivery, it
immediately becomes eligible to be retried. If you want to implement a retry
delay, you must catch exceptions and requeue the task with a delay. If your
task crashes without throwing an exception (eg SIGKILL), you could end up in
an endless retry loop that prevents other tasks from being processed.How it worksdjango-pg-queue is able to claim, process, and remove a task in a single (simplified)
query.DELETEFROMpgq_jobWHEREid=(SELECTidFROMpgq_jobWHEREexecute_at<=now()ORDERBYpriorityDESC,created_atFORUPDATESKIPLOCKEDLIMIT1)RETURNING*;As soon as this query runs, the task is unable to be claimed by other workers.
When the transaction commits, the task will be deleted. If the transaction
rolls back or the worker crashes, the task will immediately become available
for another worker.To achieve at-least-once delivery, we begin a transaction, process the task,
then commit the transaction. For at-most-once, we claim the task and
immediately commit the transaction, then process the task. For tasks that don’t
have any external effects and only do database work, the at-least-once behavior
is actually exactly-once (because both the claiming of the job and the database
work will commit or rollback together).Comparison to Celerydjango-pg-queue fills the same role as Celery. You must use postgres as the backend
and the library is small enough that you can read and understand all the code.A note on the use ofAtLeastOnceQueueand Django’stransaction.on_commit()A failure in anon_commit()callback will not cause that job to be retried
when using anAtLeastOnceQueue(usually a job in anAtLeastOnceQueuequeue will remain in the queue if the job fails).  This is becauseon_commit()callbacks are executed after the transaction has been committed
and, for django-pg-queue, the job is removed from the queue when the transaction
commits.If you require more certainty that the code in anon_commit()callback is
executed successfully, you may need to ensure it is idempotent and call it from
within the job rather than usingon_commit().UsageRequirementsdjango-pg-queue requires Python 3, at least postgres 9.5 and at least
Django 2.1.InstallationInstall with pip:pip install django-pg-queueThen add'pgq'to yourINSTALLED_APPS. Runmanage.py migrateto
create the jobs table.Instantiate a queue object. This can go wherever you like and be named whatever
you like. For example,someapp/queue.py:frompgq.queueimportAtLeastOnceQueuequeue=AtLeastOnceQueue(tasks={# ...},queue='my-queue',notify_channel='my-queue',)You will need to import this queue instance to queue or process tasks. UseAtLeastOnceQueuefor at-least-once delivery, orAtMostOnceQueuefor
at-most-once delivery.django-pg-queue comes with a management command base class that you can
use to consume your tasks. It can be called whatever you like, for example in asomeapp/managment/commands/worker.py:frompgq.commandsimportWorkerfromsomeapp.queueimportqueueclassCommand(Worker):queue=queueThen you can runmanage.py workerto start your worker.A task function takes two arguments – the queue instance in use, and the Job
instance for this task. The function can be defined anywhere and called
whatever you like. Here’s an example:frompgq.decoratorsimporttaskfrom.queuesimportqueue@task(queue)defdebug_task(queue,job):print(job.args)Instead of using the task decorator, you can manually register it as a task.
Add it to your queue instance when it is being created:queue=AtLeastOnceQueue(tasks={'debug_task':debug_task,},queue='my-queue')The key is the task name, used to queue the task. It doesn’t have to match the
function name.To queue the task, if you used the task decorator you may:debug_task.enqueue({'some_args':0})To manually queue the task, use theenqueuemethod on your queue instance:queue.enqueue('debug_task',{'some_args':0})Assuming you have a worker running for this queue, the task will be run
immediately. The second argument must be a single json-serializeable value and
will be available to the task asjob.args.Tasks registered using the@taskdecorator will only be available on the
queue if the file in which the task is defined has been imported. If your
worker doesn’t import the file containing the@taskdecorators somewhere,
the tasks will not be available for dispatch. Importing files in theapps.pyAppConfig.ready()method will ensure that the tasks are always
available on the queue without having to import them in your worker just for
the import side effects.# Contents of someapp/apps.pyfromdjango.appsimportAppConfigclassSomeAppAppConfig(AppConfig):defready(self):# Tasks registered with @task are defined in this importimportsomeapp.tasksMultiple QueuesYou may run multiple queues and workers may each listen to a queue. You can have multiple workers
listening to the same queue too. A queue is implemented as a CharField in the database.
The queue would simply filter for jobs matching its queue name.Bulk EnqueueMany jobs can be efficiently created usingbulk_enqueue()which accepts one
task name for all the jobs being created and a list of dictionaries containingargsfor the task to execute with and, optionally,priorityandexecute_atfor that particular job.queue.bulk_enqueue('debug_task',[{'args':{'some_args':0}},{'args':{'some_args':10}'priority':10,'execute_at':timezone.now()+timedelta(days=1),},])MonitoringTasks are just database rows stored in thepgq_jobtable, so you can
monitor the system with SQL.To get a count of current tasks:SELECTqueue,count(*)FROMpgq_jobWHEREexecute_at<=now()GROUPBYqueueThis will include both tasks ready to process and tasks currently being
processed. To see tasks currently being processed, we need visibility into
postgres row locks. This can be provided by thepgrowlocks extension.  Once
installed, this query will count currently-running tasks:SELECTqueue,count(*)FROMpgrowlocks('pgq_job')WHERE'For Update'=ANY(modes)GROUPBYqueue;You could join the results ofpgrowlockswithpgq_jobto get the full
list of tasks in progress if you want.Loggingdjango-pg-queue logs through Python’s logging framework, so can be
configured with theLOGGINGdict in your Django settings. It will not log
anything under the default config, so be sure to configure some form of
logging. Everything is logged under thepgqnamespace. Here is an example
configuration that will log INFO level messages to stdout:LOGGING={'version':1,'root':{'level':'DEBUG','handlers':['console'],},'formatters':{'verbose':{'format':'%(levelname)s%(asctime)s%(module)s%(process)d%(thread)d%(message)s',},},'handlers':{'console':{'level':'INFO','class':'logging.StreamHandler','formatter':'verbose',},},'loggers':{'pgq':{'handlers':['console'],'level':'INFO','propagate':False,},}}It would also be sensible to log WARNING and higher messages to something like
Sentry:LOGGING={'version':1,'root':{'level':'INFO','handlers':['sentry','console'],},'formatters':{'verbose':{'format':'%(levelname)s%(asctime)s%(module)s%(process)d%(thread)d%(message)s',},},'handlers':{'console':{'level':'INFO','class':'logging.StreamHandler','formatter':'verbose',},'sentry':{'level':'WARNING','class':'raven.contrib.django.handlers.SentryHandler',},},'loggers':{'pgq':{'level':'INFO','handlers':['console','sentry'],'propagate':False,},},}You could also log to a file by using the built-inlogging.FileHandler.Useful RecipesThese recipes aren’t officially supported features ofdjango-pg-queue. We provide them so that you can mimick some of the common features in other task queues.CELERY_ALWAYS_EAGERCelery uses theCELERY_ALWAYS_EAGERsetting to run a task immediately, without queueing it for a worker. It could be used during tests, and while debugging in a development environment with any workers turned off.classEagerAtLeastOnceQueue(AtLeastOnceQueue):defenqueue(self,*args,**kwargs):job=super().enqueue(*args,**kwargs)ifsettings.QUEUE_ALWAYS_EAGER:self.run_job(job)returnjob