Package: celery_once
Celery OnceCelery Once allows you to prevent multiple execution and queuing ofcelerytasks.InstallationInstallingcelery_onceis simple with pip, just run:pip install -U celery_onceRequirementsCelery. Built to run with Celery 4.0. Older versions may work, but are not officially supported.UsageTo usecelery_once, your tasks need to inherit from anabstractbase task calledQueueOnce.Once installed, you’ll need to configure a few options aONCEkey in celery’s conf.fromceleryimportCeleryfromcelery_onceimportQueueOncefromtimeimportsleepcelery=Celery('tasks',broker='amqp://guest@localhost//')celery.conf.ONCE={'backend':'celery_once.backends.Redis','settings':{'url':'redis://localhost:6379/0','default_timeout':60*60}}@celery.task(base=QueueOnce)defslow_task():sleep(30)return"Done!"The exact configuration, depends on which locking backend you want to use. SeeBackends.Behind the scenes, this overridesapply_asyncanddelay. It does not affect calling the tasks directly.When running the task,celery_oncechecks that no lock is in place (against a Redis key).
If it isn’t, the task will run as normal. Once the task completes (or ends due to an exception) the lock will clear.
If an attempt is made to run the task again before it completes anAlreadyQueuedexception will be raised.example.delay(10)example.delay(10)Traceback(mostrecentcalllast):..AlreadyQueued()result=example.apply_async(args=(10))result=example.apply_async(args=(10))Traceback(mostrecentcalllast):..AlreadyQueued()gracefulOptionally, instead of raising anAlreadyQueuedexception, the task can returnNoneifonce={'graceful':True}is set in the task’soptionsor when run throughapply_async.fromcelery_onceimportAlreadyQueued# Either catch the exception,try:example.delay(10)exceptAlreadyQueued:pass# Or, handle it gracefully at run time.result=example.apply(args=(10),once={'graceful':True})# or by default.@celery.task(base=QueueOnce,once={'graceful':True})defslow_task():sleep(30)return"Done!"keysBy defaultcelery_oncecreates a lock based on the task’s name and its arguments and values.
Take for example, the following task below…@celery.task(base=QueueOnce)defslow_add(a,b):sleep(30)returna+bRunning the task with different arguments will default to checking against different locks.slow_add(1,1)slow_add(1,2)If you want to specify locking based on a subset, or no arguments you can adjust the keyscelery_oncelooks at in the task’soptionswithonce={'keys':[..]}@celery.task(base=QueueOnce,once={'keys':['a']})defslow_add(a,b):sleep(30)returna+bexample.delay(1,1)# Checks if any tasks are running with the `a=1`example.delay(1,2)Traceback(mostrecentcalllast):..AlreadyQueued()example.delay(2,2)@celery.task(base=QueueOnce,once={'keys':[]})defslow_add(a,b):sleep(30)returna+b# Will enforce only one task can run, no matter what arguments.example.delay(1,1)example.delay(2,2)Traceback(mostrecentcalllast):..AlreadyQueued()timeoutAs a fall back,celery_oncewill clear a lock after 60 minutes.
This is set globally in Celery’s configuration withONCE_DEFAULT_TIMEOUTbut can be set for individual tasks using…@celery.task(base=QueueOnce,once={'timeout':60*60*10})deflong_running_task():sleep(60*60*3)unlock_before_runBy default, the lock is removed after the task has executed (using celery’safter_return). This behaviour can be changed setting the task’s optionunlock_before_run. When set toTrue, the lock will be removed just before executing the task.Caveats:Any retry of the task won’t re-enable the lock!This can only be set when defining the task, it cannot be passed dynamically toapply_async@celery.task(base=QueueOnce,once={'unlock_before_run':True})defslow_task():sleep(30)return"Done!"BackendsRedis BackendRequires:Redisis used as a distributed locking mechanism. Behind the scenes, it use redis-py’sshared, distributed Lock.Configuration:backend-celery_once.backends.Redissettingsdefault_timeout- how many seconds after a lock has been set before it should automatically timeout (defaults to 3600 seconds, or 1 hour).url- should point towards a running Redis instance (defaults toredis://localhost:6379/0). See below for the format options supportedblocking(boolean value: defaultFalse) - If set toTrue, scheduling a task (by.delay/.apply_async) will block for X seconds to acquire the lock (see:blocking_timeoutbelow). If no lock could be acquired after X seconds, will raise anAlreadyQueuedexception. This is a very specific use-case scenario and by default is disabled.blocking_timeout(int or float value: default1) - How many seconds the task will block trying to aquire the lock, ifblockingis set toTrue. Setting this toNoneset’s no timeout (equivalent to infinite seconds).The URL parser supports three patterns of urls:redis://host:port[/db][?options]: redis over TCPrediss://host:port[/db][?options]: redis over TCP with SSL enabled.redis+socket:///path/to/redis.sock[?options]: redis over a UNIX socketTheoptionsquery args are mapped to theStrictRediskeyword args.
Examples:
*redis://localhost:6379/1redis://localhost:6379/1?ssl=truerediss://localhost:6379/1redis+socket:///var/run/redis/redis.sock?db=1Example Configuration:Minimal:celery.conf.ONCE={'backend':'celery_once.backends.Redis','settings':{'url':'redis://localhost:6379/0','default_timeout':60*60}}Advanced:
Scheduling tasks blocks up to 30 seconds trying to acquire a lock before raising an exception.celery.conf.ONCE={'backend':'celery_once.backends.Redis','settings':{'url':'redis://localhost:6379/0','default_timeout':60*60,'blocking':True,'blocking_timeout':30}}File BackendConfiguration:backend-celery_once.backends.Filesettingslocation- directory where lock files will be located. Default is temporary directory.default_timeout- how many seconds after a lock has been set before it should automatically timeout (defaults to 3600 seconds, or 1 hour).Example Configuration:celery.conf.ONCE={'backend':'celery_once.backends.File','settings':{'location':'/tmp/celery_once','default_timeout':60*60}}Flask IntergrationTo avoidRuntimeError: Working outside of application contexterrors when usingcelery_oncewithFlask, you need to make theQueueOncetask base class application context aware.
If you’ve implemented Celery following the Flaskdocumentationyou can extend it like so.defmake_celery(app):celery=Celery(app.import_name,backend=app.config['CELERY_RESULT_BACKEND'],broker=app.config['CELERY_BROKER_URL'])celery.conf.update(app.config)classContextTask(celery.Task):def__call__(self,*args,**kwargs):withapp.app_context():returnself.run(*args,**kwargs)celery.Task=ContextTask# Make QueueOnce app context aware.classContextQueueOnce(QueueOnce):def__call__(self,*args,**kwargs):withapp.app_context():returnsuper(ContextQueueOnce,self).__call__(*args,**kwargs)# Attach to celery object for easy access.celery.QueueOnce=ContextQueueOncereturnceleryNow, when instead of importing theQueueOncebase, you can use the context aware base on theceleryobject.celery=make_celery(app)@celery.task(base=celery.QueueOnce)defexample_task(value):returnCustom BackendIf you want to implement a custom locking backend, seeBACKEND_GUIDE.rst.SupportTests are run against Python 2.7, 3.4 and 3.5. Other versions may work, but are not officially supported.ContributingContributions are welcome, and they are greatly appreciated! Seecontributing
guidefor more details.