Package: unipipe
unipipeUnifiedpipeline library.:warning: Experimental :warning:Build batch pipelines in Python that run anywhere -- on your laptop, on the server, and in the cloud.Easily scale local experiments to the cloud without any changesSave time by only writing each pipeline onceSave money by only paying for the compute infrastructure you needAboutunipipemakes it easy to build batch pipelines in Python, then run them either locally or in the cloud. It was originally created for machine learning workflows, but it works for any batch data processing pipeline.InstallFrom PyPI:# Minimal installpipinstallunipipe# With additional executors (e.g. 'docker', 'vertex')pipinstallunipipe[vertex]From source:# Minimal installpipinstall"unipipe @ git+ssh://git@github.com/fkodom/unipipe.git"# With additional executors (e.g. 'docker', 'vertex')pipinstall[vertex]"unipipe @ git+ssh://git@github.com/fkodom/unipipe.git"If you'd like to contribute, install all dependencies and pre-commit hooks:# Install all dependenciespipinstall"unipipe[all] @ git+ssh://git@github.com/fkodom/unipipe.git"# Setup pre-commit hookspre-commitinstallGetting StartedBuild a pipeline once using theunipipeDSL:fromunipipeimportdsl@dsl.componentdefsay_hello(name:str)->str:returnf"Hello,{name}!"@dsl.pipelinedefpipeline():say_hello(name="world")Then, run the pipeline using any of the supported backends:fromunipipeimportrunrun(# Supported executors include:#   'python' --> runs in the current Python process#   'docker' --> runs each component in a separate Docker container#   'vertex' --> runs in GCP through Vertex, which in turn uses KFPexecutor="python",pipeline=pipeline(),)Expected output:INFO:root:[say_hello-1603ae3e]-Hello,world!Run Any Python ScriptOr scaleanyPython script to the cloud using theunipipeCLI:# Same choices of executors as above.unipiperun-script\--executorvertex\--pipeline-root"gs://bucket-name/artifact-root/ \./examples/ex01_hello_world.pyThis makes experimentation easy.unipipewill automatically compose your script into a pipeline, and launch it with your chosen executor.See this example for more details.More ExamplesLinkDescriptionHello WorldCreate/run your firstunipipepipelineHello PipelineCreate pipelines with multiple stepsMulti-output ComponentsBuild components that return more than one type-checked valuePipeline ArgumentsMake pipelines reusable with dynamic inputsDependency ManagementInstall and use other Python packages in your pipelinesHardware SpecsRequest hardware (CPUs, Memory, GPUs) for your pipeline runsNested PipelinesCall existing pipelines from inside another pipelineControl FlowAdd conditional control flow to your pipelinesAdvanced Control FlowBest practices for advanced control flowPrivate DependenciesUsing private Python packagesRun Any Python ScriptRun any Python script usingunipipeWhyunipipe?unipipewas designed to mitigate issues with Kubeflow Pipelines (KFP).Kubeflow and KFP are often used by machine learning engineers to orchestrate training jobs, data preprocessing, and other computationally intensive tasks.KFP pipelines only run on Kubeflow.Kubeflow requires specialized knowledge and additional compute resources. It can be expensive and/or impractical for individuals and small teams.Managed, serverless platforms like Vertex (Google Cloud) exist, which automate all of that. But still, pipelines only run on KFP/Vertex -- not on your laptop.Why write the same pipeline twice?KFP developers often write multiple pipeline scripts. One for their laptop, and another for the cloud.TODO: Finish this section...TODOAdd executor for KFP clusters, in addition to Vertex.Better up-front type checking (in progress).