Package: django-partisan
django-backgroud-tasks-frameworkFramework to allow creating background tasks in django without MQUsageRequirementsPython versions: 3.6+Django versions: 1.9+General usageInstall app with:$ pip install django-partisanAdd it to yourINSTALLED_APPSRun migrate command:$ python manage.py migrateWrite your task and register it:# partisan_tasks.pyfromdjango_partisan.processorimportBaseTaskProcessorfromdjango_partisanimportregistry@registry.registerclassMyProcessor(BaseTaskProcessor):PRIORITY=5# Optional, by default 10UNIQUE_FOR_PARAMS=True# Optional, by default Truedefrun(self):do_something(*self.args,**self.kwargs)Please note, that all tasks should be in modulepartisan_tasks.pyin root of your application.
You can write them there or just import from your preferable place.
It's needed to make autodiscover availableAnd then put this task to queue:frompartisan_tasksimportMyProcessordefdo_something_and_postpone_task(*args,**kwargs):MyProcessor(args,kwargs).delay()To start tasks processing you should start workers:$ python manage.py start_partisanAdvancedPostpone tasksYou can postpone task by raising special exceptionPostponeTask:Also you can manage posponing delay by settingDEFAULT_POSTPONE_DELAY_SECONDSin yoursettings.py(by default - 5 seconds) or pass it toPostponeTask:fromdjango_partisan.exceptionsimportPostponeTaskdefdo_something():raisePostponeTask(300)# 5 minutesIt's possible to configure postponing rules for every single processor.
Just definePOSTPONE_CONFIGin your processor class as instance ofPostponeConfig:fromdjango_partisan.config.processor_configsimportPostponeConfigfromdjango_partisan.processorimportBaseTaskProcessorfromdjango_partisanimportregistryfromdjango_partisan.exceptionsimportPostponeTask@registry.registerclassMyProcessor(BaseTaskProcessor):POSTPONE_CONFIG=PostponeConfig(max_postpones=5)defrun(self):do_something(*self.args,**self.kwargs)defdo_something(*args,**kwargs):raisePostponeTask(300)# 5 minutesWith such configuration the task will be postponed for 5 times and, if it will be tried to be postponed one more time,MaxPostponesReachedexception will be rised.Also you can globally set maximum postpones in settings withDEFAULT_POSTPONES_COUNTinsettings.py(by default - 15).
It was made to make task processing finite. If you want to make your task be processed forever, until they would be finished,
you can set it toNone, but it is dangerous.Errors handlingYou can set for task processor special config, that is managing, how to handle errors:fromdjango_partisan.config.processor_configsimportErrorsHandleConfigfromdjango_partisan.configimportconstfromdjango_partisan.processorimportBaseTaskProcessorfromdjango_partisanimportregistry@registry.registerclassMyProcessor(BaseTaskProcessor):RETRY_ON_ERROR_CONFIG=ErrorsHandleConfig(retry_on_errors=[TimeoutError,],retries_count=3,retry_pause=3,retry_pause_strategy=const.DELAY_STRATEGY_INCREMENTAL)defrun(self):do_something(*self.args,**self.kwargs)defdo_something(*args,**kwargs):passWith such configuration, the task will be redelayed ifTimeoutErrorwill be rised for 3 times with 3 sec pause.ErrorsHandleConfigparams:retry_on_errors- list of exceptions;retries_count- positive int. Task will be redelayed forretries_counttimes if any of errors will be rised;retry_pause- positive int. Time in seconds to wait before renew task processing;retry_pause_strategy- one of options:django_partisan.config.const.DELAY_STRATEGY_INCREMENTAL,django_partisan.config.const.DELAY_STRATEGY_CONSTANT. By default -DELAY_STRATEGY_CONSTANT. If is set toDELAY_STRATEGY_CONSTANT-
on error task will be redelayed withretry_pauseseconds gap every time. If is set toDELAY_STRATEGY_INCREMENTAL-
every next time task will be redelayed with increasing byretry_pausetime gap
(withretry_pause = 3, andretries_count = 3it will redelay for 3, 6, 9 seconds and then fail).Separate by queuesIf you want to separate your tasks into separate queues, you need to define queues in setting as a dict,
where key is a queue name, and value - dict of settings for this queue:PARTISAN_CONFIG={'default':{# 'default' - is registered name for default queue, it will be exist anyway# ...},'another_queue':{# ...}}Also you need to set queue for your tasks:fromdjango_partisan.processorimportBaseTaskProcessorfromdjango_partisanimportregistry@registry.registerclassMyProcessor(BaseTaskProcessor):QUEUE_NAME='another_queue'defrun(self):do_something(*self.args,**self.kwargs)defdo_something(*args,**kwargs):passAfter it you can run partisan to run this queue:$pythonmanage.pystart_partisan--queue_nameanother_queueNote:If you will not setQUEUE_NAMEforProcessor, it will bedefault;If you will run this command without specifiingqueue_nameit will servedefaultqueue;If you will not set settings for queues, the settings will be default fordefaultqueue;SettingsIn your project settings you can define such params as:MIN_QUEUE_SIZE(int)- if number of tasks in queue will reach this amount,
queue manager will add new tasks (default = 2);MAX_QUEUE_SIZE(int)- queue manager will fill queue up to this count of tasks (default = 10);CHECKS_BEFORE_CLEANUP(int)- number of gatherings new tasks before cleaning database connections (default = 50);WORKERS_COUNT(int)- number of workers, that would be instanced by workers manager (default = number of cores);SLEEP_DELAY_SECONDS(int)- time to sleep between queue checks (default = 2);TASKS_PER_WORKER_INSTANCE(Optional[int])- if is set, the worker will be restarted after this count of
tasks processed (default = None);DELETE_TASKS_ON_COMPLETE(bool)- if True, task object will be deleted from db, if it successfully processed;But it will be better, if you'll make settings as a dict:PARTISAN_CONFIG={'default':{# 'default' - is registered name for default queue, it will be exist anyway'MIN_QUEUE_SIZE':10,'MAX_QUEUE_SIZE':20,'CHECKS_BEFORE_CLEANUP':50,'WORKERS_COUNT':5,'SLEEP_DELAY_SECONDS':5,'TASKS_PER_WORKER_INSTANCE':50,'DELETE_TASKS_ON_COMPLETE':False,'DEFAULT_POSTPONE_DELAY_SECONDS':5,'DEFAULT_POSTPONES_COUNT':None,}}It is possible to override some of this settings by cli args forstart_partisan:--min_queue_size-MIN_QUEUE_SIZE;--max_queue_size-MAX_QUEUE_SIZE;--checks_before_cleanup-CHECKS_BEFORE_CLEANUP;--workers_count-WORKERS_COUNT;--sleep_delay_seconds-SLEEP_DELAY_SECONDS;APIBaseTaskProcessorBaseTaskProcessor.delay(*, priority: int = 0, execute_after: datetime = None)accept only keyword arguments.
It is possible to override priority of task and set execution datetime (task will not be processed before this time);BaseTaskProcessor.PRIORITY- property of TaskProcessor. The lower the number, the higher the priority.
Tasks with higher priority would be taken for processing first;BaseTaskProcessor.UNIQUE_FOR_PARAMS- boolean property of TaskProcessor. IfTrue, it will ignore for
task adding if task with exactly same args and kwargs is already in queue;Some behavior featuresThis tool works only with PostgreSQL, as it supportsJSONFieldAfter Manager process got a kill signal, it will wait for workers to finish their jobs, and gracefully shut down them;If for some reason Manager process was killed without gracefull shut down,
after restart it will take for work tasks, that were not finished and only after them it will take all other tasks;Known issuesThis tool can be launched on MacOS, but it's strongly recomened to use it only with Linux as multiprocessing Queue
not fully compatible with MacOS. Use it on MacOS only for development and debugging.After calling SIG_TERM, tool can cause some exceptions. It does't break anything and I'm trying to fix it.
Don't worry about itContributingIf you experience any problems with usage of this package,
feel free to open an issue or pull-requestPull-request requirements:Code style according to PEP-8 (usemake blackfor it);100% test coverage (usemake coverage);Typing (usemake check_mypy;Version bump (you can usemake major_release,make minor_release,make patch_releasefor it)All this checks are in CI