Package: sinbadflow
Simple pipeline creation and execution toolSinbadflow is a simple pipeline creation and execution tool. It was created having Databricks notebooks workflow in mind, however with flexible implementation options the tool can be customized to fit any task. Named after famous cartoon "Sinbad: Legend of the Seven Seas" the library provides ability to create and run agents with specific triggers and conditional functions in parallel or single mode. With the simple, yet intuitive, code based syntax we can create elaborative pipelines to help with any data engineering, data science or software development task.InstallationTo install use:pip install sinbadflowUsageSinbadflow supports single or parallel run with different execution triggers. To build a pipeline use>>symbol between two agents. Example Databricks notebooks pipeline (one-by-one execution):fromsinbadflow.agents.databricksimportDatabricksAgentasdbrfromsinbadflowimportTriggerpipeline=dbr('/path/to/notebook1')>>dbr('path/to/another/notebook')Parallel run pipeline (agents in list are executed in parallel mode):pipeline=[dbr('/parallel_run_notebook'),dbr('/another_parallel_notebook')]The flow can be controlled by using triggers. Sinbadflow supports these triggers:Trigger.DEFAULT- default trigger, the agent is always executed.Trigger.OK_PREV- agent will be executed if previous agent finished successfully.Trigger.OK_ALL- agent will be executed if so far no fails are recorded in the pipeline.Trigger.FAIL_PREV- agent will be executed if previous agent run failed.Trigger.FAIL_ALL- agent will be executed if all previous runs failed.An example workflow would look like this:execution=dbr('/execute')parallel_handles=[dbr('/handle_ok',Trigger.OK_PREV),dbr('/handle_fail',Trigger.FAIL_PREV)]save=dbr('/save_all',Trigger.OK_ALL)fail_handling=dbr('/log_all_failed',Trigger.FAIL_ALL)pipeline=execution>>parallel_handles>>save>>fail_handlingTo run the pipeline:fromsinbadflowimportSinbadflowsf=Sinbadflow()sf.run(pipeline)The pipeline will be executed and results will be logged with selected method (print/loggingsupported). Sinbadflow will always run the full pipeline, there is no implementation for early stoppage if the pipeline fails.Conditional functionsFor more flexible workflow control Sinbadflow also supports conditional functions check. This serves as more elaborative triggers for the agents.fromsinbadflow.agents.databricksimportDatabricksAgentasdbrfromdatetimeimportdatedefis_monday():returndate.today().weekday()==0notebook1=dbr('/notebook1',conditional_func=is_monday)notebook2=dbr('/notebook2',conditional_func=is_monday)pipeline=notebook1>>notebook2In the example above notebooks will be skipped if today is not Monday because ofconditional_fucfunction. Sinbadflow also provides ability to apply conditional function to the whole pipeline usingapply_conditional_funcmethod.fromsinbadflow.utilsimportapply_conditional_funcpipeline=dbr('/notebook1')>>dbr('/notebook2')>>dbr('/notebook3')pipeline=apply_conditional_func(pipeline,is_monday)Custom AgentsSinbadflow provides ability to create your own agents. In order to do that, your agent must inherit fromBaseAgentclass, pass thedataandtriggerparameters to parent class (also**kwargsif you are planning to use conditional functions) and implementrun()method. An exampleDummyAgent:fromsinbadflow.agentsimportBaseAgentfromsinbadflowimportTriggerfromsinbadflowimportSinbadflowclassDummyAgent(BaseAgent):def__init__(self,data=None,trigger=Trigger.DEFAULT,**kwargs):super(DummyAgent,self).__init__(data,trigger,**kwargs)defrun(self):print(f'        Running my DummyAgent with data:{self.data}')defcondition():returnFalsesecret_data=DummyAgent('secret_data')parallel_data=[DummyAgent('simple_data',conditional_func=condition),DummyAgent('important_data',Trigger.OK_ALL)]pipeline=secret_data>>parallel_datasf=Sinbadflow()sf.run(pipeline)DatabricksAgent - cluster modesOut of the box Sinbadflow comes withDatabricksAgentwhich can be used to run Databricks notebooks on interactive or job clusters.DatabricksAgentinit arguments:notebook_path#Notebook location in the workspacetrigger=Trigger.DEFAULT#Triggertimeout=1800#Notebook run timeoutargs={}#Notebook argumentscluster_mode='interactive'#Cluster mode (interactive/job)job_args={)#Job cluster parametersconditional_func=default_func()#Conditional functionDefaultjob_argsparameters for job cluster creation (more information about job_argssee here):{'spark_version':'6.4.x-scala2.11','node_type_id':'Standard_DS3_v2','driver_node_type_id':'Standard_DS3_v2','num_workers':1}By default the notebook will be executed on interactive cluster usingdbutilslibrary. To run notebook on separate job cluster use the following code:fromsinbadflow.agents.databricksimportDatabricksAgentasdbr,JobSubmitterfromsinbadflow.executorimportSinbadflow#set new job_argsnew_job_args={'num_workers':10,'driver_node_type_id':'Standard_DS3_v2'}job_notebook=dbr('notebook1',job_args=new_job_args,cluster_mode='job')interactive_notebook=dbr('notebook2')pipeline=job_notebook>>interactive_notebook##Access token is used for job cluster creation and notebook submissionJobSubmitter.set_access_token('<DATABRICKS ACCESS TOKEN>')sf=Sinbadflow()sf.run(pipeline)As shown in the example above you can mix and match agent runs on interactive/job clusters to achieve the optimal solution.Additional helpFull API docs can be foundhere.Use built inhelp()method for additional information about the classes and methods.Do not hesitate to contact me with any question. Pull requests are encouraged!ContributorsSpecial thank you for everyone who contributed to the project:Robertas Sys,Emilija Lamanauskaite