Package: celery-dyrygent
Celery DyrygentThis project aims to support full DAG workflow processing.
It's designed as celery extension and uses celery as an execution backend.
Celery-dyrygent is released under modified BSD license. SeelicenseWhat is it?The reasons behind this project so as the implementation details were described in the following blogposthttps://www.ovh.com/blog/doing-big-automation-with-celery/What is a DAG workflow?DAG is a shortcut for Directed Acyclic Graph.
While DAG workflow would be any combination of celery primitives:groupschainschordsCelery Dyrygent is able to process any kind of DAG workflows.Why not to use native celery stuff?Celery struggles a bit with complex workflows built from combining primitives.
The execution might be unreliable, there are a lot of corner cases where workflow might not work as desired.
Serialization of complex workflows causes memory issues.
Some of the encountered problems which aren't solved (celery 4.2.1):https://github.com/celery/celery/issues/5000https://github.com/celery/celery/issues/5286https://github.com/celery/celery/issues/5327How does it work?The whole workflow machinery works simialar tochord_unlockrepeating celery task which waits till some tasks are done (header) and then executes further tasks (body).
Celery Dyrygent introduces a workflow processor task which orchestrates an execution of a whole workflow.
Once the workflow is started the workflow processor task is repeated till the workflow execution is done or till some TTL timestamp is reached (not to repeat indefinitely).
The workflow processor schedules the execution of tasks according to their relations, retries itself, then checks if the tasks are done so the new ones can be scheduled, repeat.
That's it, the idea is quite simple.Advantagesexecution part is done by Celery, so all celery machinery with its features is available (retries, countdowns, etc.)each workflow is executed in the same wayCelery operates on simple tasks only - no nested structures which causes troubleslink error for whole workflow can be implementedfinalizing task for whole workflow can be implemented (e.g. do something always when workflow finishes)workflow execution is SOLID and RELIABLEit's possible to track progress through signals (might need to implement a new signal for each tick)DrawbacksAt the moment workflow processor doesn't pass task results from precedign tasks to following tasks (can be implemented, not implemented at the moment).Workflow processor task is doing repeating ticks (like celery chord unlock) and new tasks are scheduled only within the ticks. This may result in noticeably longer execution time of task chains (e.g. if ticks are done each 2s, next task in chain will only be each 2s)Reliable result backend has to be enabledHow to use it?Which celery versions are supported?celery 3.1.25celery 4.2.1probably any celery 4.xIntegrationInitialize workflowsYou need to register workflow processor task in your celery appfromcelery_dyrygent.tasksimportregister_workflow_processorapp=Celery()#  your celery application instanceworkflow_processor=register_workflow_processor(app)Use workflow on you celery canvasWorkflows can consume celery canvas to properly build internal relationsfromcelery_dyrygent.workflowsimportWorkflowcanvas=celery.chain()|celery.chord()#  define your canvas using native celery mechanismswf=Workflow()wf.add_celery_canvas(canvas)wf.apply_async()Workflow processor task will be scheduled holding all signatures from canvas and their relations. It will execute signatures according to their relations.Signals supportCelery Dyrygent provides additional signals which can be used e.g. for tracking workflow progress. Following signals are available:after_active_tickon_finishon_state_changeHow to use signals?When a signal is emitted all registered signal handlers are executed. In order to register signal handler you need to useWorkflow.connectfunction. See examples below. The handler is called with two parameters: workflow instance and payload (optional).Usingon_state_changesignalSignal is emitted when workflow state changes. Supported states are:INITIALRUNNINGSUCCESSFAILUREERRORHandler is called with two params:workflow instancepayload - current state of a workflowfromcelery_dyrygent.workflowsimportWorkflow@Workflow.connect('on_state_change')defhandle_state_change(workflow,payload):print("Workflow{}has new state{}".format(workflow.id,payload))Usingon_finishsignalSignal is emitted when workflow is finished (or can't move forward due to failed tasks)Handler params are:workflow instancepaylod - NoneUsingafter_active_tickSignal is emitted when workflow has scheduled new tasksHandler params are:workflow instancepayload - NoneSupport for custom dataBothWorkflowandWorkflowNodehave acustom_payloaddictionary member that can be used to store
additional data. For example, one can use those dictionnary to store some application specific
metadata....wf=Workflow()fortaskintask_list:sig=create_celery_task(task)sig.freeze()node=wf.add_signature(sig)node.custom_payload['user_id']=task.user_id...Using celery task optionsYou can define custom options for your tasks, as defined in:https://docs.celeryproject.org/en/stable/reference/celery.app.task.html#celery.app.task.Task.apply_asyncThese options may be different between the workflow task and user tasks.wf=Workflow(options={'priority':10})wf.add_celery_canvas(canvas)wf.apply_async(options={'priority':8})TODOProper documentation (e.g. sphinx)Python pip release