Package: pydflow
DFLOWDflowis a Python framework for constructing scientific computing workflows (e.g. concurrent learning workflows) employingArgo Workflowsas the workflow engine.For dflow's users (e.g. ML application developers), dflow offers user-friendly functional programming interfaces for building their own workflows. Users need not be concerned with process control, task scheduling, observability and disaster tolerance. Users can track workflow status and handle exceptions by APIs as well as from frontend UI. Thereby users are enabled to concentrate on implementing operations (OPs) and orchestrating workflows.For dflow's developers, dflow wraps on argo SDK, keeps details of computing and storage resources from users, and provides extension abilities. While argo is a cloud-native workflow engine, dflow uses containers to decouple computing logic or scheduling logic, and leverages Kubernetes to make workflows observable, reproducible and robust. Dflow is designed to be based on a distributed, heterogeneous infrastructure. With high-performance computing (HPC) clusters being a common resources in scientific computing, users can either use executor to manage HPC jobs usingDPDispatcherplugin, or use virtual node technique to uniformly manage HPC resources in the framework of Kubernetes (wlm-operator).OP template (abbr. OP) in dflow can be reused across workflows and shared among users. Dflow provides a cookie cutter recipedflow-op-cutterfor template a new OP package. Start developing an OP package at once frompipinstallcookiecuttercookiecutterhttps://github.com/deepmodeling/dflow-op-cutter.gitDflow provides a debug mode for running workflows bare-metally whose backend is implemented in dflow itself in pure Python, independent of Argo/Kubernetes. The debug mode utilizes local environment to execute OPs instead of containers by default. It implements most APIs of the default (Argo) mode in order to ensure a consistent user experience. The debug mode offer convenience for debugging or testing without container. In cases where clusters face difficulties deploying Docker or Kubernetes or have limited external access, the debug mode may also be used for production, despite less robustness and observability.Overview1.1.Basics1.1.1.Parameters and artifacts1.1.2.OP template1.1.3.Step1.1.4.WorkflowQuick Start2.1.Setup Argo Server2.2.Install dflow2.3.Run an exampleUser Guide3.1.Common3.1.1.Workflow management3.1.2.Upload artifact3.1.3.Steps3.1.4.DAG3.1.5.Conditional step, parameters and artifacts3.1.6.Produce parallel steps using loop3.1.7.Timeout3.1.8.Continue on failed3.1.9.Continue on success number/ratio of parallel steps3.1.10.Optional input artifacts3.1.11.Default value for output parameters3.1.12.Key of a step3.1.13.Resubmit a workflow3.1.14.Executor3.1.15.Submit HPC/Bohrium job via dispatcher plugin3.1.16.Submit Slurm job via virtual node3.1.17.Use resources in Kubernetes3.1.18.Important note: variable names3.1.19.Debug mode: dflow independent of Kubernetes3.1.20.Artifact storage plugins3.2.Python OP3.2.1.Slices3.2.2.Retry and error handling3.2.3.Progress3.2.4.Upload python packages for development1. Overview1.1. Basics1.1.1.  Parameters and artifactsParameters and artifacts are data stored and transferred within a workflow. Parameters are saved as text which can be displayed in the UI, while artifacts are stored as files. Parameters are passed to an OP with their values, while artifacts are passed as paths.1.1.2.  OP templateOP template (abbr. OP) serves as a fundamental building block of a workflow. It defines a particular operation to be executed given the input and the expected output. Both the input and output can be parameters and/or artifacts. The most common OP template is the container OP template. Two types of container OP templates are supported:ShellOPTemplate,PythonScriptOPTemplate.ShellOPTemplatedefines an operation by shell script and a container image where the script runs.PythonScriptOPTemplatedefines an operation by Python script and a container image.As a more Python-native category of OP templates provided by dflow,PythonOPTemplatedefines OPs in the form of Python classes or Python functions (termed class OP or function OP respectively). As Python is a weak typed
language, we impose strict type checking to Python OPs to minimize ambiguity and unexpected behaviors.For a class OP, the input and output structures of an OP are declared in the static methodsget_input_signandget_output_sign. Each of them returns a dictionary mapping from the name of a parameter/artifact to its type. The execution of the OP is defined in theexecutemethod. The types of the parameter values passed in and out should be in accord with those declared in the sign. Type checking is implemented before and after theexecutemethod. For an input/output artifact, its sign should be likeArtifact(type)wheretypecan bePath,List[Path],Dict[str, Path]ordflow.python.NestedDict[Path]. For input artifact, the OP receives a path, a list of paths or a dictionary of paths according to its sign . OP developer can directly process the file or directory at the path. For output artifact, the OP should also return a path, a list of paths or a dictionary of paths according to its sign.fromdflow.pythonimportOP,OPIO,OPIOSign,ArtifactfrompathlibimportPathimportshutilclassSimpleExample(OP):def__init__(self):pass@classmethoddefget_input_sign(cls):returnOPIOSign({"msg":str,"foo":Artifact(Path),})@classmethoddefget_output_sign(cls):returnOPIOSign({"msg":str,"bar":Artifact(Path),})@OP.exec_sign_checkdefexecute(self,op_in:OPIO,)->OPIO:shutil.copy(op_in["foo"],"bar.txt")out_msg=op_in["msg"]op_out=OPIO({"msg":out_msg,"bar":Path("bar.txt"),})returnop_outThe above example illustrates an OPSimpleExample. This operation involves copying the input artifactfooto the output artifactbarand duplicating the input parametermsgto the output parametermsg.For an function OP, the input and output structures are declared more succinctly using type annotations and execution process is defined in the function body. Type checking is implemented before and after the function as well. We recommendpython>=3.9to use this syntactic sugar. To learn more about Python Annotation, refer to thePython official howtos.fromdflow.pythonimportOP,ArtifactfrompathlibimportPathimportshutil@OP.functiondefSimpleExample(msg:str,foo:Artifact(Path),)->{"msg":str,"bar":Artifact(Path)}:shutil.copy(foo,"bar.txt")out_msg=msgreturn{"msg":out_msg,"bar":Path("bar.txt")}To create an OP template based on the above class or function, we need to specify the container image and any optional arguments toPythonOPTemplate.pydflowneed not to be installed in this image because localpydflowpackage will automatically be uploaded into the container by defaultfromdflow.pythonimportPythonOPTemplatesimple_example_templ=PythonOPTemplate(SimpleExample,image="python:3.8")An example is herePython OP example1.1.3. StepStepserves as the central component for establishing data flow rules. A step is created by instantiating an OP template, which requires specifying the values for all input parameters and sources for all input artifacts declared in the OP template. The input parameters/artifacts of a step may be either static at the time of submission, or dynamically from outputs of another step.fromdflowimportStepsimple_example_step=Step(name="step0",template=simple_example_templ,parameters={"msg":"HelloWorld!"},artifacts={"inp_art":foo},)Note thatfoohere is an artifact either uploaded locally or output of another step.1.1.4.  WorkflowWorkflowconnects steps together to build a workflow. A simple serial workflow is created by adding steps in sequence. Adding a list of steps to a workflow means these steps will run concurrently.fromdflowimportWorkflowwf=Workflow(name="hello-world")wf.add(simple_example_step)Submit a workflow bywf.submit()An example is hereWorkflow example2. Quick Start2.1. Setup Argo ServerIf you have an Argo server already, you can skip this step. Otherwise you can follow theinstallation guide.2.2. Install dflowMake sure your Python version is not less than 3.6 and install dflowpip install pydflow2.3. Run an exampleThere are severalnotebook tutorialsthat can help you start to use dflow. Besides, you can submit a simple workflow from the terminalpython examples/test_python.pyThen you can check the submitted workflow throughargo's UI.3. User Guide (dflow-doc)3.1. Common3.1.1. Workflow managementAfter submitting a workflow bywf.submit(), or retrieving a history workflow bywf = Workflow(id="xxx"), you can track its real-time status with APIswf.id: workflow ID in argowf.query_status(): query workflow status, return"Pending","Running","Succeeded", etc.wf.query_step(name=None, key=None, phase=None, id=None, type=None): query step by name (support for regex)/key/phase/ID/type, return a list of argo step objectsstep.phase: phase of a step,"Pending","Running",Succeeded, etc.step.outputs.parameters: a dictionary of output parameters mapping parameter names to parameter objects, get the value of a parameter bystep.outputs.artifacts["foo"].valuestep.outputs.artifacts: a dictionary of output artifacts mapping artifact names to artifact objects, download an artifact bydownload_artifact(step.outputs.artifacts["bar"])wf.terminate(): terminate a workflow3.1.2. Upload artifactDflow offers tools for uploading files to the artifact repository and downloading files from it (default artifact repository is Minio set up in the quick start). User can upload a file/directory, a list of files/directories or a dictionary of files/directories and obtain an artifact object, which can be used as argument of a stepartifact=upload_artifact([path1,path2])step=Step(...artifacts={"foo":artifact})Modifydflow.s3_configto configure artifact repository settings globally.Note: during the upload process, dflow retains the relative path of the uploaded file/directory with respect to the current directory. If file/directory outside current directory is uploaded, its absolute path is used as the relative path in the artifact. If you want a different directory structure in the artifact compared to the local one, you can create soft links and then upload.3.1.3. StepsStepsis another type of OP template which is defined by its constituent steps instead of a container. It can be viewed as a sub-workflow or a super OP template composed of smaller OPs. Astepsincludes an array of arrays ofsteps, e.g.[[s00,s01],[s10,s11,s12]], where the inner array represents concurrent steps while the outer array indicates sequential steps. Input/output parameters/artifacts for a steps can be declared as follows:steps.inputs.parameters["msg"]=InputParameter()steps.inputs.artifacts["foo"]=InputArtifact()steps.outputs.parameters["msg"]=OutputParameter()steps.outputs.parameters["bar"]=OutputArtifact()Adding astepto astepsis similar to adding one to aworkflowsteps.add(step)stepscan be used as the template to instantiate a biggerstepjust like script OP templates. This allows for constructing complex workflows of nested structure. It is also possible to recursively use astepsas the template of a building block within itself to achieve dynamic loop.Recursive exampleTo set an output parameter ofstepsfrom a step within itsteps.outputs.parameters["msg"].value_from_parameter=step.outputs.parameters["msg"]Here,stepmust be contained withinsteps. Similarly, to assign an output artifact forsteps, usesteps.outputs.artifacts["foo"]._from=step.outputs.parameters["foo"]3.1.4. DAGDAGis another type of OP template which is defined by its constituent tasks and their dependencies. The usage ofDAGis similar to that ofSteps. To incorporate ataskinto adag, usedag.add(task)The usage oftaskis also similar to that ofstep. Dflow will automatically detect dependencies among tasks of adagbased on their input/output relations. Additional dependencies can be specified bytask_3=Task(...,dependencies=[task_1,task_2])DAG example3.1.5. Conditional step, parameters and artifactsSet a step to be conditional byStep(..., when=expr)whereexpris an boolean expression in string format. Such as"%s < %s" % (par1, par2). The step will be executed if the expression is evalutated to be true, otherwise skipped. Thewhenargument is often used as the breaking condition of recursive steps. The output parameter of asteps(similar todag) can be assigned as conditional bysteps.outputs.parameters["msg"].value_from_expression=if_expression(_if=par1<par2,_then=par3,_else=par4)Similarly, the output artifact of astepscan be assigned as conditional bysteps.outputs.artifacts["foo"].from_expression=if_expression(_if=par1<par2,_then=art1,_else=art2)Conditional outputs example3.1.6. Produce parallel steps using loopIn scientific computing, it is often required to produce a list of parallel steps which share a common OP template, and only differ in the input parameters.with_paramandwith_sequenceare 2 arguments ofStepfor automatically generating a list of parallel steps. These steps share a common OP template, and only differ in the input parameters.A step usingwith_paramoption generates parallel steps based on a list (either a constant list or referring to another parameter, e.g. an output parameter of another step or an input parameter of thestepsorDAGcontext), the parallelism equals to the length of the list. Each parallel step picks an item from the list by"{{item}}", such asstep=Step(...parameters={"msg":"{{item}}"},with_param=steps.inputs.parameters["msg_list"])A step usingwith_sequenceoption generates parallel steps based on a numeric sequence.with_sequenceis usually used in coordination withargo_sequencewhich returns an Argo's sequence. Forargo_sequence, the number at which to start the sequence is specified bystart(default: 0). You can either specify the number of elements in the sequence bycountor the ending number of the sequence byend. The printf format string can be specified byformatto format the values in the sequence. Each argument can be passed with a parameter,argo_lenwhich returns the length of a list may be useful. Each parallel step picks an element from the sequence by"{{item}}", such asstep=Step(...parameters={"i":"{{item}}"},with_sequence=argo_sequence(argo_len(steps.inputs.parameters["msg_list"])))3.1.7. TimeoutSet the timeout of a step byStep(..., timeout=t). The unit is second.Timeout example3.1.8. Continue on failedTo allow a workflow to continue even if a step fails byStep(..., continue_on_failed=True).Continue on failed example3.1.9. Continue on success number/ratio of parallel stepsFor a group of parallel steps generated bywith_paramorwith_sequence, enable the workflow to continue when a specific number/ratio of parallel steps succeed byStep(..., continue_on_num_success=n)orStep(..., continue_on_success_ratio=r).Continue on success ratio example3.1.10. Optional input artifactsSet an input artifact to be optional byop_template.inputs.artifacts["foo"].optional = True.3.1.11. Default value for output parametersSet default value for an output parameter byop_template.outputs.parameters["msg"].default = default_value. The default value will be used when the expression invalue_from_expressionfails or the step is skipped.3.1.12. Key of a stepYou can assign a key to a step byStep(..., key="some-key")for the convenience of locating the step. The key can be regarded as an input parameter which may contain references to other parameters. For instance, the key of a step may change with iterations of a dynamic loop. Once a key is assigned to a step, the step can be query bywf.query_step(key="some-key"). If the key is unique within the workflow, thequery_stepmethod returns a singleton list (a list consist of only one element).Key of step example3.1.13. Resubmit a workflowWorkflows often include some computationally expensive steps. Outputs of previously run steps can be reused for submitting a new workflow. This allows, e.g. a failed workflow to be restarted from a specific point after modifying the workflow template or even the outputs of completed steps. For example, submit a workflow with reused steps bywf.submit(reuse_step=[step0, step1]). Here,step0andstep1are previously run steps returned byquery_stepmethod. Before the new workflow runs a step, it will detect if there exists a reused step with a matching key. If a match is found, the workflow will skip the step and use the outputs from the reused step. To modify the outputs of a step before reusing, usestep0.modify_output_parameter(par_name, value)for parameters andstep0.modify_output_artifact(art_name, artifact)for artifacts.Reuse step example3.1.14. ExecutorFor a container/script step (a step using script OP template), by default the Shell script or Python script runs in the container directly. Alternatively, you can modify the executor to run the script differently. Dflow offers an extension point for container/script stepStep(..., executor=my_executor). Here,my_executorshould be an instance of class derived from the abstract classExecutor. An implementation class ofExecutorshould implement a methodrenderwhich converts original template to a new template.classExecutor(ABC):@abc.abstractmethoddefrender(self,template):passA context is similar to an executor, but assigned to a workflowWorkflow(context=...), affecting every step.3.1.15. Submit HPC/Bohrium job via dispatcher pluginDPDispatcheris a python package used to generate HPC scheduler systems (Slurm/PBS/LSF) orBohriumjobs input scripts, submit these scripts and poke until they finish. Dflow provides a simple interface to utilize dispatcher as an executor to accomplish script steps. E.g.fromdflow.plugins.dispatcherimportDispatcherExecutorStep(...,executor=DispatcherExecutor(host="1.2.3.4",username="myuser",queue_name="V100"))For SSH authentication, you can either specify the path of a private key file locally, or upload an authorized private key to each node (or equivalently add each node to the authorized host list). For configuring additionalmachine, resources or task parameters for dispatcher, useDispatcherExecutor(..., machine_dict=m, resources_dict=r, task_dict=t).Dispatcher executor example3.1.16. Submit Slurm job via virtual nodeFollowing the installation steps in thewlm-operatorproject to add Slurm partitions as virtual nodes to Kubernetes (use manifestsconfigurator.yaml,operator-rbac.yaml,operator.yamlin this project which modified some RBAC configurations)$ kubectl get nodes
NAME                            STATUS   ROLES                  AGE    VERSION
minikube                        Ready    control-plane,master   49d    v1.22.3
slurm-minikube-cpu              Ready    agent                  131m   v1.13.1-vk-N/A
slurm-minikube-dplc-ai-v100x8   Ready    agent                  131m   v1.13.1-vk-N/A
slurm-minikube-v100             Ready    agent                  131m   v1.13.1-vk-N/AThen you can assign a step to be executed on a virtual node (i.e. submit a Slurm job to the corresponding partition to complete the step)step=Step(...executor=SlurmJobTemplate(header="#!/bin/sh\n#SBATCH --nodes=1",node_selector={"kubernetes.io/hostname":"slurm-minikube-v100"}))3.1.17. Use resources in KubernetesA step can also be completed using a Kubernetes resource (e.g. Job or custom resources). At the beginning, a manifest is applied to Kubernetes. Then the status of the resource is monitered until the success condition or the failure condition is satisfied.classResource(ABC):action=Nonesuccess_condition=Nonefailure_condition=None@abc.abstractmethodpassWlm example3.1.18. Important note: variable namesDflow has following restrictions on variable names.Variable nameStatic/DynamicRestrictionsExampleWorkflow/OP template nameStaticLowercase RFC 1123 subdomain (must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character), not more than 57 characters longmy-nameStep/Task nameStaticMust consist of alpha-numeric characters or '-', and must start with an alpha-numeric characterMy-name1-2, 123-NAMEParameter/Artifact nameStaticMust consist of alpha-numeric characters, '_' or '-'my_param_1, MY-PARAM-1Key nameDynamicLowercase RFC 1123 subdomain (must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character)my-name3.1.19. Debug mode: dflow independent of KubernetesThe debug mode is enabled by settingfromdflowimportconfigconfig["mode"]="debug"Before running a workflow locally, ensure that the dependencies of all OPs within the workflow are properly configured in the locally environment. This is not necessary when using the dispatcher executor to submit jobs to some remote environments. The debug mode uses the current directory as the working directory by default. Each workflow will generate a new directory there, whose structure will be likepython-lsev6
├── status
└── step-penf5
    ├── inputs
    │   ├── artifacts
    │   │   ├── dflow_python_packages
    │   │   ├── foo
    │   │   └── idir
    │   └── parameters
    │       ├── msg
    │       └── num
    ├── log.txt
    ├── outputs
    │   ├── artifacts
    │   │   ├── bar
    │   │   └── odir
    │   └── parameters
    │       └── msg
    ├── phase
    ├── script
    ├── type
    └── workdir
        ├── ...The top level contains the the workflow's status and all its steps. The directory name for each step will be its key if provided, or generated from its name otherwise. Each step directory contains the input/output parameters/artifacts, type and phase of the step. For a step of type "Pod", its directory also includes the script, log file and working directory for the step.Debug mode examples3.1.20. Artifact storage pluginsThe default storage for artifacts in dflow is a Minio deployment in the Kubernetes cluster. While other artifact storages are supported (e.g. Aliyun OSS, Azure blob storage (ABS), Google cloud storage(GCS)). Dflow also offers an extension point for implementing customized storage in the artifact management. This can be achieved through a storage clientStorageClient, a class implementing 5 abstract methods,upload,download,list,copyandget_md5(optional), which offer the functionality of uploading file, downloading file, listing files with specific prefix, copying file on the server side and getting the MD5 sum of file, respectively. Use a custom storage client object by configuring s3_config["storage_client"].classStorageClient(ABC):@abc.abstractmethoddefupload(self,key:str,path:str)->None:pass@abc.abstractmethoddefdownload(self,key:str,path:str)->None:pass@abc.abstractmethoddeflist(self,prefix:str,recursive:bool=False)->List[str]:pass@abc.abstractmethoddefcopy(self,src:str,dst:str)->None:pass@abc.abstractmethoddefget_md5(self,key:str)->str:pass3.2. Python OP3.2.1. SlicesIn coordination withparallel steps,Slicesassists user to slice input parameters/artifacts (which must be lists) to feed parallel steps and stack their output parameters/artifacts into lists following the same pattern. The Python OP only need to handle one slice. For example,step=Step(name="parallel-tasks",template=PythonOPTemplate(...,slices=Slices(input_parameter=["msg"],input_artifact=["data"],output_artifact=["log"])),parameters={"msg":msg_list},artifacts={"data":data_list},with_param=argo_range(5))In this example, each item inmsg_listis passed to a parallel step as the input parametermsg, each part indata_listis passed to a parallel step as the input artifactdata. Finally, the output artifactslogof all parallel steps are collected to a single artifactstep.outputs.artifacts["log"].
This example is analogous to the following pseudocodelog=[None]*5foriteminrange(5):log[item]=my_op(msg=msg_list[item],data=data_list[item])wherewith_paramandslicescorresponds to theforloop and the statement in loop in the pseudocode, respectively.Slices exampleIt should be noticed that this feature, by default, passes full input artifacts to each parallel step which may only use some slices of these artifacts. In contrast, the subpath mode of slices only passes one single slice of the input artifacts to each parallel step. To use the subpath mode of slices,step=Step(name="parallel-tasks",template=PythonOPTemplate(...,slices=Slices(sub_path=True,input_parameter=["msg"],input_artifact=["data"],output_artifact=["log"])),parameters={"msg":msg_list},artifacts={"data":data_list})Here, each input parameter and artifact to be sliced must be of the same length, and the parallelism equals to this length. Another notable point is that in order to use the subpath of the artifacts, these artifacts must be stored without compression when they are generated. E.g. declareArtifact(..., archive=None)in the output signs of Python OP, or specifyupload_artifact(..., archive=None)while uploading artifacts. Besides, you can usedflow.config["archive_mode"] = Noneto set default archive mode to no compression globally.Subpath mode of slices example3.2.2. Retry and error handlingDflow catchesTransientErrorandFatalErrorthrown fromOP. User can set maximum number of retries onTransientErrorbyPythonOPTemplate(..., retry_on_transient_error=n). Timeout error is regarded as fatal error for default. To treat timeout error as transient error, setPythonOPTemplate(..., timeout_as_transient_error=True). When a fatal error is raised or the retries on transient error reaches maximum retries, the step is considered as failed.Retry example3.2.3. ProgressAOPcan update progress in the runtime so that user can moniter its real-time progress from the frontend UIclassProgress(OP):progress_total=100...defexecute(op_in):foriinrange(10):self.progress_current=10*(i+1)...Progress example3.2.4. Upload python packages for developmentTo avoid frequently creating images during development, dflow offers an interface to upload local packages into the container and add them to$PYTHONPATH, such asPythonOPTemplate(..., python_packages=["/opt/anaconda3/lib/python3.9/site-packages/numpy"]). Additionally, you can globally specify packages to be uploaded, which will impact allOPsfromdflow.pythonimportupload_packagesupload_packages.append("/opt/anaconda3/lib/python3.9/site-packages/numpy")