Project: Couler (Provisioning / Automation & Configuration)
File: README.md

<img width="300px" src="./assets/logo.svg" alt="Couler Logo"> ](https://github.com/couler-proj/couler/actions?query=event%3Apush+branch%3Amaster) ](https://twitter.com/CoulerProject) Couler aims to provide a unified interface for constructing and managing workflows on different workflow engines, such as , , and . Couler is included in and . You can find a list of organizations who are using Couler in . If you'd like to add your organization to the list, please send us a pull request. Many workflow engines exist nowadays, e.g. , , and . However, their programming experience varies and they have different level of abstractions that are often obscure and complex. The code snippets below are some examples for constructing workflows using Apache Airflow and . === "Apache Airflow" ```python def createdag(dagid, schedule, dagnumber, defaultargs): def helloworldpy(*args): print('Hello World') dag = DAG(dag_id, schedule_interval=schedule, defaultargs=defaultargs) with dag: t1 = PythonOperator( taskid='helloworld', pythoncallable=helloworld_py, dagnumber=dagnumber) return dag for n in range(1, 10): default_args = {'owner': 'airflow', 'start_date': datetime(2018, 1, 1) } globals()[dagid] = createdag( 'helloworld{}'.format(str(n)), '@daily', n, default_args) ``` === "Kubeflow Pipelines" ```python class FlipCoinOp(dsl.ContainerOp): """Flip a coin and output heads or tails randomly.""" def init(self): super(FlipCoinOp, self).init( name='Flip', image='python:alpine3.6', command=['sh', '-c'], arguments=['python -c "import random; result = \'heads\' if random.randint(0,1) == 0 ' 'else \'tails\'; print(result)" | tee /tmp/output'], file_outputs={'output': '/tmp/output'}) class PrintOp(dsl.ContainerOp): """Print a message.""" def init(self, msg): super(PrintOp, self).init( name='Print', image='alpine:3.6', command=['echo', msg], ) @graph_component def flipcomponent(flipresult): printflip = PrintOp(flipresult) flipA = FlipCoinOp().after(print_flip) with dsl.Condition(flipA.output == 'heads'): flip_component(flipA.output) @dsl.pipeline( name='pipeline flip coin', description='shows how to use graph_component.' ) def recursive(): flipA = FlipCoinOp() flipB = FlipCoinOp() fliploop = flipcomponent(flipA.output) flip_loop.after(flipB) PrintOp('cool, it is over. %s' % flipA.output).after(flip_loop) ``` Couler provides a unified interface for constructing and managing workflows that provides the following: Simplicity: Unified interface and imperative programming style for defining workflows with automatic construction of directed acyclic graph (DAG). Extensibility: Extensible to support various workflow engines. Reusability: Reusable steps for tasks such as distributed training of machine learning models. Efficiency: Automatic workflow and resource optimizations under the hood. Please see the following sections for installation guide and examples. To set up Couler and run your first workflow, please see . For more examples of Couler usage, please see . ptimization in cloud and the contributions are outlined below:: Simplicity and Extensibility: Couler provides a unified programming interface for workflow definition, ensuring independence from the workflow engine and compatibility with various workflow engines such as Argo Workflows, Airflow, and Tekton. Automation: Couler integrates LLMs in unified programming code generation. By leveraging LLMs, Couler facilitates the generation of unified programming code using NL descriptions. Additionally, we automate hyperparameters tuning through the integration of Dataset Card and Model Card, enhancing the effectiveness of the autoML process. Efficiency: Couler introduces the Intermediate Representative (IR) to depict the workflow Directed Acyclic Graph (DAG), optimizing extensive workflow computations by dividing a large workflow into smaller ones for auto-parallelism optimization. Couler also implements dynamic caching of artifacts, which are the outputs of jobs in the workflow, to minimize redundant computations and ensure fault

Open Source Community: The released open-source version of Couler has garnered adoption from multiple companies and end-users. For instance, over 3000 end users are utilizing Couler within Ant Group, and more than 20 companies have adopted Couler as their default workflow engine interface. Please see the following sections for installation guide and examples. Couler currently only supports Argo Workflows. Please see instructions to install Argo Workflows on your Kubernetes cluster. Install Python 3.6+ Install Couler Python SDK via the following command: ```bash python3 -m pip install git+https://github.com/couler-proj/couler --ignore-installed ``` Alternatively, you can clone this repository and then run the following to install: ```bash python setup.py install ``` Click to launch the interactive Katacoda environment and learn how to write and submit your first Argo workflow using Couler Python SDK in your browser! This example combines the use of a Python function result, along with conditionals, to take a dynamic path in the workflow. In this example, depending on the result of the first step defined in `flip_coin()`, the template will either run the `heads()` step or the `tails()` step. Steps can be defined via either `couler.run_script()` for Python functions or `couler.run_container()` for containers. In addition, the conditional logic to decide whether to flip the coin in this example is defined via the combined use of `couler.when()` and `couler.equal()`. ```python import couler.argo as couler from couler.argo_submitter import ArgoSubmitter def random_code(): import random res = "heads" if random.randint(0, 1) == 0 else "tails" print(res) def flip_coin(): return couler.runscript(image="python:alpine3.6", source=randomcode) def heads(): return couler.run_container( image="alpine:3.6", command=["sh", "-c", 'echo "it was heads"'] ) def tails(): return couler.run_container( image="alpine:3.6", command=["sh", "-c", 'echo "it was tails"'] ) result = flip_coin() couler.when(couler.equal(result, "heads"), lambda: heads()) couler.when(couler.equal(result, "tails"), lambda: tails()) submitter = ArgoSubmitter() couler.run(submitter=submitter) ``` This example demonstrates different ways to define the workflow as a directed-acyclic graph (DAG) by specifying the dependencies of each task via `couler.set_dependencies()` and `couler.dag()`. Please see the code comments for the specific shape of DAG that we've defined in `linear()` and `diamond()`. ```python import couler.argo as couler from couler.argo_submitter import ArgoSubmitter def job(name): couler.run_container( image="docker/whalesay:latest", command=["cowsay"], args=[name], step_name=name, ) def linear(): couler.set_dependencies(lambda: job(name="A"), dependencies=None) couler.set_dependencies(lambda: job(name="B"), dependencies=["A"]) couler.set_dependencies(lambda: job(name="C"), dependencies=["A"]) couler.set_dependencies(lambda: job(name="D"), dependencies=["B"]) def diamond(): couler.dag( [ [lambda: job(name="A")], [lambda: job(name="A"), lambda: job(name="B")], # A -> B [lambda: job(name="A"), lambda: job(name="C")], # A -> C [lambda: job(name="B"), lambda: job(name="D")], # B -> D [lambda: job(name="C"), lambda: job(name="D")], # C -> D ] ) linear() submitter = ArgoSubmitter() couler.run(submitter=submitter) ``` Note that the current version only works with Argo Workflows but we are actively working on the design of the unified interface that is extensible to additional workflow engines. Please stay tuned for more updates and we welcome any feedback and contributions from the community. Please cite the repo if you use the code in this repo. ```bibtex @misc{Couler, author = {Xiaoda Wang, Yuan Tang, Tengda Guo, Bo Sang, Jingji Wu, Jian Sha, Ke Zhang, Jiang Qian, Mingjie Tang}, title = {Couler: Unified Machine Learning Workflow Optimization in Cloud}, year = {2024}, publisher = {40th IEEE International Conference on Data Engineering}, howpublished = {\url{https://arxiv.org/abs/2403.07608}}. } ```