Package: pipefunc
pipefunc: function composition magic for PythonLightweight function pipeline creation: 📚 Less Bookkeeping, 🎯 More Doing:books: Table of Contents:thinking: What is this?:rocket: Key Features:test_tube: How does it work?:notebook: Jupyter Notebook Example:computer: Installation:hammer_and_wrench: Development:thinking: What is this?pipefuncis a Python library designed to create and manage complex networks of interdependent functions, often known as function pipelines.In a function pipeline, each function can have dependencies on the results of other functions. Managing these dependencies, ensuring each function has the inputs it needs, and determining the order of execution can become an annoying bookkeeping task in complex cases.pipefuncsimplifies this process by allowing you to declare the dependencies of each function and automatically organizing the execution order to satisfy these dependencies. Additionally, the library provides features for visualizing the function pipeline, simplifying the pipeline graph, caching function results for efficiency, and profiling resource usage for optimization.For example, imagine you have a set of functions wherefunction Bneeds the output fromfunction A, andfunction Cneeds the outputs from bothfunction Aandfunction B.pipefuncallows you to specify these dependencies when you create the functions and then automatically manages their execution. It also provides tools for visualizing this function network, simplifying it if possible, and understanding the resource usage of each function.The library is designed to be an efficient and flexible tool for managing complex function dependencies in an intuitive and clear way. Whether you're dealing with data processing tasks, scientific computations, machine learning (AI) workflows, or other scenarios where functions depend on one another,pipefunccan help streamline your code and improve your productivity.:rocket: Key FeaturesSome of the key features ofpipefuncinclude:🚀Function Composition and Pipelining:The core functionality ofpipefuncis to create a pipeline of functions, allowing you to feed the output of one function into another, and execute them in the right order.📊Visualizing Pipelines:pipefunccan generate a visual graph of the function pipeline, making it easier to understand the flow of data.💡Flexible Function Arguments:pipefunclets you call a function with different combinations of arguments, automatically determining which other functions to call based on the arguments you provide.👥Multiple Outputs:pipefuncsupports functions that return multiple results, allowing each result to be used as input to other functions.➡️Reducing Pipelines:pipefunccan simplify a complex pipeline by merging nodes, improving computational efficiency at the cost of losing visibility into some intermediate steps.🎛️Resources Report:pipefuncprovides a report on the performance of your pipeline, including CPU usage, memory usage, and execution time, helping you identify bottlenecks and optimize your code.🔄Parallel Execution and Caching:pipefuncsupports parallel execution of functions, and caching of results to avoid redundant computation.🔍Parameter Sweeps:pipefuncprovides a utility for generating combinations of parameters to use in a parameter sweep, along with the ability to cache results to optimize the sweep.🛠️Flexibility and Ease of Use:pipefuncis a lightweight, flexible, and powerful tool for managing complex function dependencies in a clear and intuitive way, designed to improve your productivity in any scenario where functions depend on one another.:test_tube: How does it work?pipefunc provides a Pipeline class that you use to define your function pipeline.
You add functions to the pipeline using thepipefuncdecorator, which also lets you specify a function's output name and dependencies.
Once your pipeline is defined, you can execute it for specific output values, simplify it by combining functions with the same root arguments, visualize it as a directed graph, and profile the resource usage of the pipeline functions.
For more detailed usage instructions and examples, please check the usage example provided in the package.Here is a simple example usage of pipefunc to illustrate its primary features:frompipefuncimportpipefunc,Pipeline# Define three functions that will be a part of the pipeline@pipefunc(output_name="c")deff_c(a,b):returna+b@pipefunc(output_name="d")deff_d(b,c):returnb*c@pipefunc(output_name="e")deff_e(c,d,x=1):returnc*d*x# Create a pipeline with these functionsfuncs=[f_c,f_d,f_e]pipeline=Pipeline(funcs,profile=True)# You can access and call these functions using the func methodh_d=pipeline.func("d")asserth_d(a=2,b=3)==15h_e=pipeline.func("e")asserth_e(a=2,b=3,x=1)==75asserth_e(c=5,d=15,x=1)==75# Visualize the pipelinepipeline.visualize()# Get all possible argument mappings for each functionall_args=pipeline.all_arg_combinations()print(all_args)# Show resource reporting (only works if profile=True)pipeline.resources_report()This example demonstrates defining a pipeline withf_c,f_d,f_efunctions, accessing and executing these functions using the pipeline, visualizing the pipeline graph, getting all possible argument mappings, and reporting on the resource usage.
This basic example should give you an idea of how to use pipefunc to construct and manage function pipelines.:notebook: Jupyter Notebook ExampleSee the detailed usage example and more in ourexample.ipynb.:computer: InstallationInstall thelatest stableversion from conda (recommended):condainstall"pipefunc[plotting]"or from PyPI:pipinstallpipefuncor installmainwith:pipinstall-Uhttps://github.com/basnijholt/pipefunc/archive/main.zipor clone the repository and do a dev install (recommended for dev):gitclonegit@github.com:basnijholt/pipefunc.gitcdpipefunc
pipinstall-e".[dev,test,plotting]":hammer_and_wrench: DevelopmentWe usepre-committo manage pre-commit hooks, which helps us ensure that our code is always clean and compliant with our coding standards.
To set it up, install pre-commit with pip and then run the install command:pipinstallpre-commit
pre-commitinstall