Package: sca_rhythm
RhythmRhythm allows you to design and control workflows made of Celery tasks. A workflow is a sequence of steps to run one
after the other. Rhythm simplifies the process of executing workflows consisting of long-running tasks with reliability.The following are the features of Rhythm workflows:If a workflow consisting of three steps (S1, S2, and S3) encounters a failure while executing S2 (even after retries
by Celery), it is possible to resume the workflow later. Resuming the workflow with restart S2 with previous arguments
and after its completion S3 will be run.A workflow can be paused and resumed later.You can keep track of which step is currently running, as well as its progress.Installationpip install sca-rhythmsee onpypiPrerequisitesCelery app should be configured with a mongo database backend.Create Tasks withWorkflowTaskclassimportosimporttimefromceleryimportCeleryfromsca_rhythmimportWorkflowTaskapp=Celery("tasks")@app.task(base=WorkflowTask,bind=True)deftask1(self,batch_id,**kwargs):print(f'task -{os.getpid()}1 starts with{batch_id}')# do worktime.sleep(1)# update progress to result backend# sets the task's state as "PROGRESS"self.update_progress({'done':2873,'total':100000})# do some more workreturnbatch_id,{'return_obj':'foo'}:warning: Task Constraints :warning:The task signature must contain**kwargsfor the workflow orchestration to function.The return type must be of list / tuple type and the first element of the return value is sent to the next task as
its argument.Create Workflows withWorkflowclassfromceleryimportCeleryfromsca_rhythmimportWorkflowsteps=[{'name':'inspect','task':'tasks.inspect''queue':'q1',},{'name':'archive','task':'tasks.archive','queue':'q1',},{'name':'stage','task':'tasks.stage','queue':'q2','priority':5}]wf=Workflow(app,steps=steps,name='archive_batch',app_id='app')wf.start('batch-id-test')The provided code defines a workflow consisting of multiple steps, each representing a task to be executed in a specific order. The workflow is initiated with a unique identifier, and its steps are configured with task names, associated queues, and optional priorities.Each step is represented as a dictionary with the following properties:name: A descriptive name for the step.task: The task to be executed, specified as a string containing the task's import path.queue: The Celery queue to which the task should be sent.priority (optional): An integer (between 0 and 9) indicating the priority of the task in the queue. If not provided, the priority is set to the step's position in the workflow. If there are more than 9 tasks, tasks in positions 10 and above will all recieve priority 9.Priority Scheme:
The priority scheme is designed to optimize the execution of tasks within the same workflow. Tasks with higher priorities are executed before those with lower priorities. If no priority is specified, the default priority is set to the step's position in the workflow. This scheme ensures that tasks within a workflow are executed sequentially with increasing priority, minimizing the likelihood of interweaving tasks from different workflows.Pause and Resume WorkflowsPausing a workflow stop the current running task and resuming a workflow will restart the stopped task with the same
arguments.wf=Workflow(app,workflow_id='2f87decb-a431-472b-b26e-32c894993881')wf.pause()wf.resume()Build & Publishpoetryinstall
poetrypublish--buildTask StatusPENDING: Task state is unknown (assumed pending since you know the id).STARTED: Task was started by a worker (task_track_started = True)SUCCESS: Task succeededFAILURE: Task failedREVOKED: Task was revokedRETRY: Task is waiting for retry.Workflow StatusThe workflow status is a summative status that is determined by the status of the initial step that is not marked as "
SUCCESS," which is referred to as a "pending step".PENDING - the pending step is the first step in the workflow and its status is pending.STARTED - the status of the pending step is one of STARTED, RETRY, PENDING.REVOKED - the pending step was revoked, the workflow can be resumed.FAILURE - the pending step was failed, the workflow can be resumed.SUCCESS - all steps have succeeded.Status Groups:DONE: { SUCCESS, FAILURE, REVOKED }ACTIVE: !DONE