Package: django-lightweight-queue
Django Lightweight QueueDLQ is a lightweight & modular queue and cron system for Django. It powers
millions of production jobs every day at Thread.Installationpipinstalldjango-lightweight-queue[redis]Currently the only production-ready backends are redis-based, so theredisextra is essentially required. Additional non-redis backed production-ready
backends are great candidates for community contributions.Basic UsageStart by addingdjango_lightweight_queueto yourINSTALLED_APPS:INSTALLED_APPS=["django.contrib.admin","django.contrib.auth",...,"django_lightweight_queue",]After that, define your task in any file you want:importtimefromdjango_lightweight_queueimporttask# Define a task@task()deflong_running_task(first_arg,second_arg):time.sleep(first_arg*second_arg)# Request that the task be executed at some pointlong_running_task(4,second_arg=9)See the docstring on thetaskdecorator
for more details.ConfigurationAll automatically picked up configuration options begin withLIGHTWEIGHT_QUEUE_and can be found inapp_settings.py. They should be placed in the usual Django
settings files, for example:LIGHTWEIGHT_QUEUE_BACKEND='django_lightweight_queue.backends.redis.RedisBackend'Special ConfigurationIf desired, specific configuration overrides can be placed in a standalone
python file which passed on the command line. This is useful for applying
customisations for specific servers.For example, given aspecial.pycontaining:LIGHTWEIGHT_QUEUE_REDIS_PORT=12345and then running:$ python manage.py queue_runner --extra-settings=special.pywill result in the runner to use the settings from the specified configuration
file in preference to settings from the Django environment. Any settings not
present in the specified file are inherited from the global configuration.BackendsThere are four built-in backends:Synchronous (Development backend)django_lightweight_queue.backends.synchronous.SynchronousBackendExecutes the task inline, without any actual queuing.Redis (Production backend)django_lightweight_queue.backends.redis.RedisBackendExecutes tasks at-most-once usingRedisfor storage of the enqueued tasks.Reliable Redis (Production backend)django_lightweight_queue.backends.reliable_redis.ReliableRedisBackendExecutes tasks at-least-once usingRedisfor storage of the enqueued tasks (subject to Redis consistency). Does not guarantee the taskcompletes.Debug Web (Debug backend)django_lightweight_queue.backends.debug_web.DebugWebBackendInstead of running jobs it prints the url to a view that can be used to run a task in a transaction which will be rolled back. This is useful for debugging and optimising tasks.Use this to append the appropriate URLs to the bottom of your rooturls.py:fromdjango.confimportsettingsfromdjango.urlsimportpath,includeurlpatterns=[...]ifsettings.DEBUG:urlpatterns+=[path("",include("django_lightweight_queue.urls",namespace="django-lightweight-queue"),)]This backend may require an extra setting if your debug site is not on localhost:# defaults to http://localhost:8000LIGHTWEIGHT_QUEUE_SITE_URL="http://example.com:8000"Running WorkersThe queue runner is implemented as a Django management command:$ python manage.py queue_runnerWorkers can be distributed over multiple hosts by telling each runner that it is
part of a pool:$ python manage.py queue_runner --machine 2 --of 4Alternatively a runner can be told explicitly how to behave by having
extra settings loaded (anyLIGHTWEIGHT_QUEUE_*constants found in the file
will replace equivalent django settings) and being configured to run exactly as
the settings describe:$ python manage.py queue_runner --exact-configuration --extra-settings=special.pyWhen using--exact-configurationthe number of workers is configured exactly,
rather than being treated as the configuration for a pool. Additionally,
exactly-configured runners willnotrun any cron workers.ExampleGiven a Django configuration containing:LIGHTWEIGHT_QUEUE_WORKERS={'queue1':3,}and aspecial.pycontaining:LIGHTWEIGHT_QUEUE_WORKERS={'queue1':2,}Running any of:$ python manage.py queue_runner --machine 1 --of 3 # or,
$ python manage.py queue_runner --machine 2 --of 3 # or,
$ python manage.py queue_runner --machine 3 --of 3will result in one worker forqueue1on the current machine, while:$ python manage.py queue_runner --exact-configuration --extra-settings=special.pywill result in two workers on the current machine.Cron TasksDLQ supports the use of a cron-like specification of Django management commands
to be run at certain times.To specify that a management command should be run at a given time, place acron.pyfile in the root folder of the Django app which defines the command
and which contains aCONFIGvariable:CONFIG=({'command':'my_cron_command',# Day values 1-7 to match datetime.datetime.utcnow().isoweekday()'days':'*','hours':'*','minutes':'*',# Equivalent behaviour to the kwarg to `task` of the same name'sigkill_on_stop':True,},)MaintainersThis repository was created byChris LambatThread, and continues to be maintained by theThread
engineering team.