Package: pyperunner
PypeRunner is an easy to use yet powerful workflow pipeline tool written in pure python, with parallel processing of tasks, smart
caching of results and reproducible runs.
PypeRunner allows for the creation of complex workflows with branching and merging of several concurrent execution flows.FeaturesEasy to use extract, transform load (ETL) pipeline toolReproducible pipeline runs for data science / machine learning tasksEasy creation of pipelines using functional API chaining (see below)Parallel processing of stepsCaching of previously run steps to speed up processingRe-run of steps (and all subsequent steps) when parameters are changedSave & read pipelines to/from yamlGraphical output of pipeline runInstallationInstall from conda forge viacondainstallpyperunner-cconda-forgeInstall from pip viapipinstallpyperunnerOr from source viagitclonehttps://github.com/glichtner/pyperunner.gitcdpyperunnerpythonsetup.pyinstallQuickstartPyperunner has three basic components:Task: Definition of the work to do (Python classes or functions)Pipeline: A collection of tasks that are connected in a directed fashionRunner: The executor of a pipelineHello world examplefrompyperunnerimportRunner,Pipeline,task@task("Hello",receives_input=False)defhello():print("in hello()")return"Hello"@task("World")defworld(data):hello=data["Hello()"]print("in world()")returnf"{hello}world"# instantiate and connect taskshello=hello()world=world()(hello)# create pipeline and set root elementpipeline=Pipeline("hello-world-example",[hello])# print a summary of the pipelinepipeline.summary()# run pipelinerunner=Runner(data_path="data/",log_path="log/")runner.run(pipeline)# get pipeline results object from the pipeline that was just runresults=runner.results()# show the resultsfortask_nameinresults:print(f"Output of task '{task_name}' was '{results[task_name]}'")Running this script outputs the following:~/pyperunner/examples$ python hello-world-func.py

+---------+
| Hello() |
+---------+
      *
      *
      *
+---------+
| World() |
+---------+

2021-01-03 20:55:47 INFO     MainProcess  root       Storing pipeline parameters in examples/log/hello-world-example_210103T205547/pipeline.yaml
2021-01-03 20:55:47 INFO     MainProcess  root       Storing pipeline data in examples/data
2021-01-03 20:55:47 INFO     Process-1    Hello()    Starting
2021-01-03 20:55:47 INFO     Process-1    Hello()    in hello()
2021-01-03 20:55:47 INFO     Process-1    Hello()    Finished: Status.SUCCESS
2021-01-03 20:55:47 INFO     Process-2    World()    Starting
2021-01-03 20:55:47 INFO     Process-2    World()    in world()
2021-01-03 20:55:47 INFO     Process-2    World()    Finished: Status.SUCCESS
2021-01-03 20:55:47 INFO     MainProcess  root       Pipeline run finished

Output of task 'Hello()' was 'Hello'
Output of task 'World()' was 'Hello world'Note that if you re-run the script, pyperunner will detect that the current configuration has already run and
will skip the execution of these tasks:~/pyperunner/examples$ python hello-world.py

2021-01-03 20:56:36 INFO     MainProcess  root       No need to execute task "Hello()", skipping it
2021-01-03 20:56:36 INFO     MainProcess  root       No need to execute task "World()", skipping it
2021-01-03 20:56:36 INFO     MainProcess  root       Pipeline run finishedIf you need to reprocess outputs, just add theforce_reload=Trueparameter to the pipeline run:runner.run(pipeline,force_reload=True)Or to run just a specific task again, use thereload=Trueparameter when initializing the task:# instantiate and connect taskshello=hello()world=world(reload=True)(hello)Note that pyperunner detects which tasks it must re-execute: All depending tasks of a reloaded task are automatically
re-executed, and only those tasks are fully skipped from execution from which the output is not required in a successor
task. Also, if a task has been previously executed and its output is required, it is read from disk.~/pyperunner/examples$ python hello-world.py

2021-01-03 20:57:26 INFO     Process-1    Hello()    Starting
2021-01-03 20:57:26 INFO     Process-1    Hello()    Loading output from disk, skipping processing
2021-01-03 20:57:26 INFO     Process-1    Hello()    Finished: Status.SUCCESS
2021-01-03 20:57:26 INFO     Process-2    World()    Starting
2021-01-03 20:57:26 INFO     Process-2    World()    in world()
2021-01-03 20:57:26 INFO     Process-2    World()    Finished: Status.SUCCESS
2021-01-03 20:57:26 INFO     MainProcess  root       Pipeline run finishedAt each run, the pipeline is automatically stored in a yaml file in the log path to ensure reproducibility:pipeline:name:hello-world-exampletasks:Hello():hash:22179f3afd85ab64dd32c63bc21a9eb4module:__main__name:Helloparams:{}parents:[]tag:''World():hash:f7d904856f2aa4fda20e05521298397fmodule:__main__name:Worldparams:{}parents:-Hello()tag:''Additionally, a graphical representation of the run is saved in the log path:DocumentationTheAPI Referenceprovides API-level documentation.ExamplesLook in the examples/ directory for some example scripts.Multiple paths pipeline# Create pipelinepipeline=Pipeline("my-pipeline")# Create first stream of tasks: LoadData(csv) --> ProcessData(normalize-l2)load_db=LoadData("database",database={"host":"localhost","username":"user","password":"password"},wait=10,)norm_l2=ProcessData("normalize-l2",norm="l2",axis=0,wait=1)(load_db)# Create second stream of tasks:#  LoadData(csv) --> ProcessData(normalize-l1) --> AugmentData(augment)load_csv=LoadData("csv",filename="data.csv",wait=1)norm_l1=ProcessData("normalize-l1",norm="l1",wait=1)(load_csv)augment=AugmentData("augment",types=["rotate","noise"],wait=1)(norm_l1)# Combine outputs of both streams (ProcessData(normalize-l2)# and AugmentData(augment)), additionally add output from ProcessData(normalize-l1)evaluate=Evaluate("both",wait=1)([norm_l1,norm_l2,augment])# Add the roots of both streams to the pipelinepipeline.add(load_db)pipeline.add(load_csv)# print a summary of the pipelinepipeline.summary()# Run pipelinerunner=Runner(data_path="data/",log_path="log/")runner.run(pipeline,force_reload=False)pipeline.summary()prints the following ascii summary:+---------------+
                                                                      | LoadData(csv) |
                                                                      +---------------+
                                                                              *
                                                                              *
                                                                              *
    +--------------------+                                      +---------------------------+
    | LoadData(database) |                                      | ProcessData(normalize-l1) |
    +--------------------+                                      +---------------------------+
              *                                                    ***                  ***
              *                                                ****                        ***
              *                                              **                               ****
+---------------------------+                 +----------------------+                          ****
| ProcessData(normalize-l2) |                 | AugmentData(augment) |                   *******
+---------------------------+****             +----------------------+            *******
                                 *******                  *                *******
                                        *******          *          *******
                                               ****      *      ****
                                                +----------------+
                                                | Evaluate(both) |
                                                +----------------+Notice how multiple tasks run simultaneously:2021-01-0319:09:05INFOProcess-1LoadData(csv)Starting2021-01-0319:09:05INFOProcess-2LoadData(database)Starting2021-01-0319:09:06INFOProcess-1LoadData(csv)Finished:Status.SUCCESS2021-01-0319:09:06INFOProcess-3ProcessData(normalize-l1)Starting2021-01-0319:09:07INFOProcess-3ProcessData(normalize-l1)Finished:Status.SUCCESS2021-01-0319:09:07INFOProcess-4AugmentData(augment)Starting2021-01-0319:09:08INFOProcess-4AugmentData(augment)Finished:Status.SUCCESS2021-01-0319:09:15INFOProcess-2LoadData(database)Finished:Status.SUCCESS2021-01-0319:09:15INFOProcess-5ProcessData(normalize-l2)Starting2021-01-0319:09:16INFOProcess-5ProcessData(normalize-l2)Finished:Status.SUCCESS2021-01-0319:09:16INFOProcess-6Evaluate(both)Starting2021-01-0319:09:17INFOProcess-6Evaluate(both)Finished:Status.SUCCESSChange LogVersion History0.1.1Added LICENSE to source dist0.1.1Fixed setup.py for conda installation0.1.0PypeRunner’s first release