Package: yape
Yape is a general-purpose pipeline/workflow executor written in Python. It
provides ways to construct and run an execution graph.Yape’s features:Ergonomics of Python: The execution graph is created via Python code,
providing great flexibility on how you create and configure your execution
nodes. The API tries its best to provide an syntax-sugary way of creating
and using nodes.Graph Execution: Graph execution can be done via CLI or a Python API. You
can execute the whole graph or selected a set of target nodes for execution.Node dependencies: A node can use as dependencies a mixture of regular
python values, other nodes and file system paths. When nodes are used as
dependencies, Yape will know how to execute them in the correct order and to
provide the results to the dependent nodes.Caching: By default, if Yape sees that an execution node did not change
from its last run, it does not re-execute it and just return the cached
result. This behavior can be turned off if necessary.Minimal execution sub-graph: Yape is capable of generating and exporting a
minimal sub-graph with only the nodes/data necessary to fulfill a selected set
of target nodes. This is specially useful for machine learning pipelines into
production, where one is only interested in the execution path for the “test
phase”.InstallYape can be installed via pip:pip install yapeQuickstartCreating the execution graphBy default, theyapecommand looks for and load the fileyp.py, which is
the entry point for creating the execution graph. In general, that file would be
at the root directory of your project. In the Python code examples provided
here, unless explicitly noted, the content is expected to be in theyp.pyfile.The most common way of creating a node is by wrapping a function responsible for
the execution of such a node. There are multiple ways you can create functional
nodes:Using theyp.nodedecorator:importyapeasyp@yp.nodedefhello(who="world"):print(f"Hello,{who}!")When using@yp.node,hellois transformed into ayape.Nodeobject, which will execute the original function.Using theyp.fndecorator:importyapeasyp@yp.fndefhello(who):print(f"Hello,{who}!")defnodegen():hello("world")In this example,hellois transformed into a special kind of function
called anode generatorand we use it insidenodegen().When a node generator is called, it creates a new node object that will call
the wrapped function with the same arguments passed to the generator.The functionnodegen()is a special function that Yape looks for when
loading theyp.pyfile and is understood as the function responsible for
creating the node objects foryp.py[1].Usingyp.fndecorator directly for node creation:importyapeasypdefhello(who):print(f"Hello,{who}!")defnodegen():yp.fn(hello)("world")This has the same effect achieved with example (2), but with the difference
thathellois left untouched, which could be useful if you want to keep
your functions unmodified for other uses. This is also useful when using
functions provided by other libraries.[1]Note that in example (1) we did not create anodegen()function as the node is already created by@yp.node, but
you could have anodegen()there as well (in case you use a mixture of
node creation styles).RunningYou can use the commandyapeto run your execution graph. Using the example
from the above:$ yape
Hello, world!Ignoring the cacheIf you try running it again, you will see that there will be no output:$ yapeThat’s because the node hasn’t changed, so Yape knows it does not have to
execute it. If we change the node definition or arguments, then Yape will detect
the change. For example, let’s change the argument for our node:importyapeasypdefhello(who):print(f"Hello,{who}!")defnodegen():yp.fn(hello)("my friend")And then runyape:$ yape
Hello, my friend!The commandyapewithout positional arguments is actually a shortcut foryape run, which is the sub-command responsible for running the execution
graph. If you want to force the execution of nodes and ignore the cache, you can
use the-foption (short for--force):$ yape run -f
Hello, my friend!Selecting target nodesTheyape runsub-command also allows us to select which nodes we want to
execute. Let’s increment our example by defining extra nodes:importyapeasypdefhello(who):print(f"Hello,{who}!")defhi(who):print(f"Hi,{who}!")defnodegen():yp.fn(hello)("my friend")yp.fn(hello,name="hello_world")("world")yp.fn(hi)("John Doe")We created two extra nodes. By default, a functional node will be named after
the name of the wrapped function. Since the first node already will be named
“hello”, we explicitly define a different name (“hello_world”) for the second
one.We can select nodes to be run by passing their names (or paths when they belong
to sub-graphs) as positional arguments:$ yape run hello_world
Hello, world!$ yape run -f hello hi # Using -f because hello is cached
Hi, John Doe!
Hello, my friend!