Package: argo-workflow-tools
Argo Workflow Toolsargo-workflow-tools is a set of tools intended to easue the usage of argo for data science and data engineerign workflowsInstallationargo-workflow-tools is published to the Python Package Index (PyPI) under the name argo-workflow-tools. To install it, run:pipinstallargo-workflow-toolsArgo SubmitterArgo Submitter is an easy to use argo client that allows data scientists to easily execute and control Argo Workflows from code and interactive notebooks.Quick StartRunning workflows from templatesThe simplest way to submit a new workflow is by running a workflow from templateARGO_CLIENT='http://localhost:2746'client=ArgoClient(ARGO_CLIENT,options=ArgoOptions(client_side_validation=False,namespace='argo'))result=client.submit('test-workflow',params={'message':'hello world'})result.wait_for_completion()You can wait for template completion by settingwait=Trueparameter, or calling wait_for_completion()result=client.submit('test-workflow',params={'message':'hello world'},wait=True)You may send parameters, through the params dictionaryresult=client.submit('test-workflow',params={'message':'hello world'},wait=True)You send objects as parameters, and they will be automatically serialized to json.ARGO_CLIENT='http://localhost:2746'client=ArgoClient(ARGO_CLIENT,options=ArgoOptions(client_side_validation=False,namespace='argo'))result=client.submit('test-workflow',params={'name':{'first':'Lorne','last':'Malvo'}},wait=True)Running workflows from specificationif you have a custom workflow manifest , you can run it by usingcreateresult=client.create(workflow_manifest,wait=True)Working with workflow resultsYou can check the status of a workflow by calling the status fieldresult.statusYou can fetch output parametes and artifacts throut the output fieldprint(result.outputs['message'])As well as reach artifacts through the s3 path propertypandas.read_csv(result.outputs['users'].s3)Controlling workflowsYou may cancel a running flow through the cancel methodresult.cancel()You may ssuspend, resume or cancel your workflow at any timeresult=client.submit('test-workflow',params={'message':'hello world'},wait=False)result.suspend()...result.resume()You can retry a failing workflow through the retry methodresult.retry()Pythonic workflow DSLFargo is a library for autoring Argo Workflows in a Python and friendly way. The main goal of Hera areMake Argo Workflows accessible by leveraging pythonic style of dagAllow seamlless local runs, for debug or testing while maintaining the same codebase for running DAG's at scalepythonic DSL is an opinionate subset of writing Argo workflows, it favors simplicity, readability and "pythonic flow" over leveraging the entire capability set Argo Workflows brings.Conceptstask - atomic python codeDAG - atonic workflows codeQuick start####Hello Worlddefsay_hello(name):message=f"hello{name}"print(message)returnmessagesay_hello("Brian")to run this simple task in argo all we need to do is to decorate our code in a task decorator@dsl.Task(image="python3:3.10")defsay_hello(name:str):message=f"hello{name}"print(message)returnmessageworkflow=Workflow(name="hello-world",entrypoint=say_hello,arguments={"name":"Brian"})print(workflow.to_yaml())DAGDAG functions are functions that define a workflow by calling other tasks or nested DAGs,
We support task depndency declaration implicitly by analyzing inputs and outputs of each task.When writing DAG functions make sure you keep it a simple as possible, call only DAG or TASK flows.@dsl.Task(image="python:3.10")defmultiply_task(x:int):returnx*2@dsl.Task(image="python:3.10")defsum_task(x:int,y:int):returnx+y@dsl.DAG()defdiamond(num:int):a=multiply_task(num)b=multiply_task(a)c=multiply_task(num)returnsum_task(b,c)Explicit depndenciesIn case a task does not return a parameter, you can set an explicit dependency by sending wait_for argument to the next task@dsl.Task(image="python:3.10")defprint_task():print(f"text")@dsl.DAG()defdiamond():a=print_task()b=print_task(wait_for=a)c=print_task(wait_for=a)print_task(wait_for=[c,b])Loopswe support map reduce workflows through[for in]loop, the iterable input must be a parameter, an output of a previous task, or a sequence object.currently only one level of nesting is supported, in case you wish to use nested loops, extract the second loop into a fucntion and decorate it as well with a DAG decorater.@dsl.Task(image="python:3.10")defgenerate_list(partitions:int,partition_size:int):items=[]foriinrange(partitions):items.append(list(range(1,partition_size)))returnitems@dsl.Task(image="python:3.10")defsum_task(items:list[int]):returnsum(items)@dsl.DAG()defmap_reduce(partitions,partition_size):partition_list=generate_list(partitions,partition_size)partition_sums=[sum_task(partition)forpartitioninpartition_list]returnsum_task(partition_sums)Conditionswe support conditional task run by employing the 'with condition' syntax@Task(image="python:3.10")defsay_hello(name:str):message=f"hello{name}"print(message)returnmessage@Task(image="python:3.10")defsay_goodbye(name:str):message=f"goodbye{name}"print(message)returnmessage@DAG()defcommand_hello(name,command):withCondition().equals(command,"hello"):say_hello(name)withCondition().equals(command,"goodbye"):say_goodbye(name)How to contributeHave any feedback? Wish to implement an extenstion or new capability? Want to help us make argo better and easier to use?
Every contribution toArgo Workflow Toolsis greatly appreciated.