Package: django-rq
Django-RQDjango integration withRQ, aRedisbased Python queuing library.Django-RQis a
simple app that allows you to configure your queues in django’ssettings.pyand easily use them in your project.Support Django-RQIf you finddjango-rquseful, please consider supporting its development viaTidelift.RequirementsDjango(2.0+)RQInstallationInstalldjango-rq(ordownload from PyPI):pipinstalldjango-rqAdddjango_rqtoINSTALLED_APPSinsettings.py:INSTALLED_APPS=(# other apps"django_rq",)Configure your queues in django’ssettings.py:RQ_QUEUES={'default':{'HOST':'localhost','PORT':6379,'DB':0,'USERNAME':'some-user','PASSWORD':'some-password','DEFAULT_TIMEOUT':360,'REDIS_CLIENT_KWARGS':{# Eventual additional Redis connection arguments'ssl_cert_reqs':None,},},'with-sentinel':{'SENTINELS':[('localhost',26736),('localhost',26737)],'MASTER_NAME':'redismaster','DB':0,# Redis username/password'USERNAME':'redis-user','PASSWORD':'secret','SOCKET_TIMEOUT':0.3,'CONNECTION_KWARGS':{# Eventual additional Redis connection arguments'ssl':True},'SENTINEL_KWARGS':{# Eventual Sentinel connection arguments# If Sentinel also has auth, username/password can be passed here'username':'sentinel-user','password':'secret',},},'high':{'URL':os.getenv('REDISTOGO_URL','redis://localhost:6379/0'),# If you're on Heroku'DEFAULT_TIMEOUT':500,},'low':{'HOST':'localhost','PORT':6379,'DB':0,}}RQ_EXCEPTION_HANDLERS=['path.to.my.handler']# If you need custom exception handlersIncludedjango_rq.urlsin yoururls.py:urlpatterns+=[path('django-rq/',include('django_rq.urls'))]UsagePutting jobs in the queueDjango-RQallows you to easily put jobs into any of the queues defined insettings.py. It comes with a few utility functions:enqueue- push a job to thedefaultqueue:importdjango_rqdjango_rq.enqueue(func,foo,bar=baz)get_queue- returns anQueueinstance.importdjango_rqqueue=django_rq.get_queue('high')queue.enqueue(func,foo,bar=baz)In addition tonameargument,get_queuealso acceptsdefault_timeout,is_async,autocommit,connectionandqueue_classarguments. For example:queue=django_rq.get_queue('default',autocommit=True,is_async=True,default_timeout=360)queue.enqueue(func,foo,bar=baz)You can provide your own singleton Redis connection object to this function so that it will not
create a new connection object for each queue definition. This will help you limit
number of connections to Redis server. For example:importdjango_rqimportredisredis_cursor=redis.StrictRedis(host='',port='',db='',password='')high_queue=django_rq.get_queue('high',connection=redis_cursor)low_queue=django_rq.get_queue('low',connection=redis_cursor)get_connection- accepts a single queue name argument (defaults to “default”)
and returns a connection to the queue’s Redis server:importdjango_rqredis_conn=django_rq.get_connection('high')get_worker- accepts optional queue names and returns a newRQWorkerinstance for specified queues (ordefaultqueue):importdjango_rqworker=django_rq.get_worker()# Returns a worker for "default" queueworker.work()worker=django_rq.get_worker('low','high')# Returns a worker for "low" and "high"@job decoratorTo easily turn a callable into an RQ task, you can also use the@jobdecorator that comes withdjango_rq:fromdjango_rqimportjob@jobdeflong_running_func():passlong_running_func.delay()# Enqueue function in "default" queue@job('high')deflong_running_func():passlong_running_func.delay()# Enqueue function in "high" queueYou can pass in any arguments that RQ’s job decorator accepts:@job('default',timeout=3600)deflong_running_func():passlong_running_func.delay()# Enqueue function with a timeout of 3600 seconds.It’s possible to specify default forresult_ttldecorator keyword argument
viaDEFAULT_RESULT_TTLsetting:RQ={'DEFAULT_RESULT_TTL':5000,}With this setting, job decorator will setresult_ttlto 5000 unless it’s
specified explicitly.Running workersdjango_rq provides a management command that starts a worker for every queue
specified as arguments:python manage.py rqworker high default lowIf you want to runrqworkerin burst mode, you can pass in the--burstflag:python manage.py rqworker high default low --burstIf you need to use custom worker, job or queue classes, it is best to use global settings
(seeCustom queue classesandCustom job and worker classes). However, it is also possible
to override such settings with command line options as follows.To use a custom worker class, you can pass in the--worker-classflag
with the path to your worker:python manage.py rqworker high default low --worker-class 'path.to.GeventWorker'To use a custom queue class, you can pass in the--queue-classflag
with the path to your queue class:python manage.py rqworker high default low --queue-class 'path.to.CustomQueue'To use a custom job class, provide--job-classflag.Starting from version 2.10, running RQ’s worker-pool is also supported:python manage.py rqworker-pool default low medium --num-workers 4Support for Scheduled JobsWith RQ 1.2.0. you can usebuilt-in schedulerfor your jobs. For example:fromdjango_rq.queuesimportget_queuequeue=get_queue('default')job=queue.enqueue_at(datetime(2020,10,10),func)If you are using built-in scheduler you have to start workers with scheduler support:python manage.py rqworker --with-schedulerAlternatively you can useRQ Scheduler.
After install you can also use theget_schedulerfunction to return aSchedulerinstance for queues defined in settings.py’sRQ_QUEUES.
For example:importdjango_rqscheduler=django_rq.get_scheduler('default')job=scheduler.enqueue_at(datetime(2020,10,10),func)You can also use the management commandrqschedulerto start the scheduler:python manage.py rqschedulerSupport for django-redis and django-redis-cacheIf you havedjango-redisordjango-redis-cacheinstalled, you can instruct django_rq to use the same connection information
from your Redis cache. This has two advantages: it’s DRY and it takes advantage
of any optimization that may be going on in your cache setup (like using
connection pooling orHiredis.)To use configure it, use a dict with the keyUSE_REDIS_CACHEpointing to the
name of the desired cache in yourRQ_QUEUESdict. It goes without saying
that the chosen cache must exist and use the Redis backend. See your respective
Redis cache package docs for configuration instructions. It’s also important to
point out that since the django-redis-cacheShardedClientsplits the cache
over multiple Redis connections, it does not work.Here is an example settings fragment fordjango-redis:CACHES={'redis-cache':{'BACKEND':'redis_cache.cache.RedisCache','LOCATION':'localhost:6379:1','OPTIONS':{'CLIENT_CLASS':'django_redis.client.DefaultClient','MAX_ENTRIES':5000,},},}RQ_QUEUES={'high':{'USE_REDIS_CACHE':'redis-cache',},'low':{'USE_REDIS_CACHE':'redis-cache',},}Queue Statisticsdjango_rqalso provides a dashboard to monitor the status of your queues at/django-rq/(or whatever URL you set in yoururls.pyduring installation.You can also add a link to this dashboard link in/adminby addingRQ_SHOW_ADMIN_LINK = Trueinsettings.py. Be careful though, this will
override the default admin template so it may interfere with other apps that
modifies the default admin template.These statistics are also available in JSON format via/django-rq/stats.json, which is accessible to staff members.
If you need to access this view via other
HTTP clients (for monitoring purposes), you can defineRQ_API_TOKENand access it via/django-rq/stats.json/<API_TOKEN>.Note: Statistics of scheduled jobs display jobs fromRQ built-in scheduler,
not optionalRQ scheduler.Additionally, these statistics are also accessible from  the command line.pythonmanage.pyrqstatspythonmanage.pyrqstats--interval=1# Refreshes every secondpythonmanage.pyrqstats--json# Output as JSONpythonmanage.pyrqstats--yaml# Output as YAMLConfiguring SentrySentry
should be configured within the Djangosettings.pyas described in theSentry docs.You can override the default Django Sentry configuration when running therqworkercommand
by passing thesentry-dsnoption:./manage.py rqworker--sentry-dsn=https://*****@sentry.io/222222This will override any existing Django configuration and reinitialise Sentry,
setting the following Sentry options:{'debug':options.get('sentry_debug'),'ca_certs':options.get('sentry_ca_certs'),'integrations':[RedisIntegration(),RqIntegration(),DjangoIntegration()]}Configuring LoggingRQ uses Python’slogging, this means you can easily configurerqworker’s logging mechanism in django’ssettings.py. For example:LOGGING={"version":1,"disable_existing_loggers":False,"formatters":{"rq_console":{"format":"%(asctime)s%(message)s","datefmt":"%H:%M:%S",},},"handlers":{"rq_console":{"level":"DEBUG","class":"rq.logutils.ColorizingStreamHandler","formatter":"rq_console","exclude":["%(asctime)s"],},},'loggers':{"rq.worker":{"handlers":["rq_console","sentry"],"level":"DEBUG"},}}Custom Queue ClassesBy default, every queue will useDjangoRQclass. If you want to use a custom queue class, you can do so
by adding aQUEUE_CLASSoption on a per queue basis inRQ_QUEUES:RQ_QUEUES={'default':{'HOST':'localhost','PORT':6379,'DB':0,'QUEUE_CLASS':'module.path.CustomClass',}}or you can specifyDjangoRQto use a custom class for all your queues inRQsettings:RQ={'QUEUE_CLASS':'module.path.CustomClass',}Custom queue classes should inherit fromdjango_rq.queues.DjangoRQ.If you are using more than one queue class (not recommended), be sure to only run workers
on queues with same queue class. For example if you have two queues defined inRQ_QUEUESand
one has custom class specified, you would have to run at least two separate workers for each
queue.Custom Job and Worker ClassesSimilarly to custom queue classes, global custom job and worker classes can be configured usingJOB_CLASSandWORKER_CLASSsettings:RQ={'JOB_CLASS':'module.path.CustomJobClass','WORKER_CLASS':'module.path.CustomWorkerClass',}Custom job class should inherit fromrq.job.Job. It will be used for all jobs
if configured.Custom worker class should inherit fromrq.worker.Worker. It will be used for running
all workers unless overridden byrqworkermanagement commandworker-classoption.Testing TipFor an easier testing process, you can run a worker synchronously this way:fromdjango.testimportTestCasefromdjango_rqimportget_workerclassMyTest(TestCase):deftest_something_that_creates_jobs(self):...# Stuff that init jobs.get_worker().work(burst=True)# Processes all jobs then stop....# Asserts that the job stuff is done.Synchronous ModeYou can set the optionASYNCtoFalseto make synchronous operation the
default for a given queue. This will cause jobs to execute immediately and on
the same thread as they are dispatched, which is useful for testing and
debugging. For example, you might add the following after you queue
configuration in your settings file:# ... Logic to set DEBUG and TESTING settings to True or False ...# ... Regular RQ_QUEUES setup code ...ifDEBUGorTESTING:forqueueConfiginRQ_QUEUES.values():queueConfig['ASYNC']=FalseNote that setting theis_asyncparameter explicitly when callingget_queuewill override this setting.Running TestsTo rundjango_rq’s test suite:`which django-admin` test django_rq --settings=django_rq.tests.settings --pythonpath=.Deploying on UbuntuCreate an rqworker service that runs the high, default, and low queues.sudo vi /etc/systemd/system/rqworker.service[Unit]Description=Django-RQWorkerAfter=network.target[Service]WorkingDirectory=<<path_to_your_project_folder>>
ExecStart=/home/ubuntu/.virtualenv/<<your_virtualenv>>/bin/python \
    <<path_to_your_project_folder>>/manage.py\rqworkerhighdefaultlow[Install]WantedBy=multi-user.targetEnable and start the servicesudosystemctlenablerqworkersudosystemctlstartrqworkerDeploying on HerokuAdddjango-rqto yourrequirements.txtfile with:pipfreeze>requirements.txtUpdate yourProcfileto:web:gunicorn--pythonpath="$PWD/your_app_name"config.wsgi:applicationworker:pythonyour_app_name/manage.pyrqworkerhighdefaultlowCommit and re-deploy. Then add your new worker with:herokuscaleworker=1ChangelogSeeCHANGELOG.md.