Package: znflow
ZnFlowTheZnFlowpackage provides a basic structure for building computational
graphs based on functions or classes. It is designed as a lightweight
abstraction layer tolearn graph computing.build your own packages on top of it.InstallationpipinstallznflowUsageConnecting FunctionsWith ZnFlow you can connect functions to each other by using the@nodifydecorator. Inside theznflow.DiGraphthe decorator will return aFunctionFutureobject that can be used to connect the function to other nodes.
TheFunctionFutureobject will also be used to retrieve the result of the
function. Outside theznflow.DiGraphthe function behaves as a normal
function.importznflow@znflow.nodifydefcompute_mean(x,y):return(x+y)/2print(compute_mean(2,8))# >>> 5withznflow.DiGraph()asgraph:mean=compute_mean(2,8)graph.run()print(mean.result)# >>> 5withznflow.DiGraph()asgraph:n1=compute_mean(2,8)n2=compute_mean(13,7)n3=compute_mean(n1,n2)graph.run()print(n3.result)# >>> 7.5Connecting ClassesIt is also possible to connect classes. They can be connected either directly or
via class attributes. This is possible by returningznflow.Connectionsinside
theznflow.DiGraphcontext manager. Outside theznflow.DiGraphthe class
behaves as a normal class.In the following example we use a dataclass, but it works with all Python
classes that inherit fromznflow.Node.importznflowimportdataclasses@znflow.nodifydefcompute_mean(x,y):return(x+y)/2@dataclasses.dataclassclassComputeMean(znflow.Node):x:floaty:floatresults:float=Nonedefrun(self):self.results=(self.x+self.y)/2withznflow.DiGraph()asgraph:n1=ComputeMean(2,8)n2=compute_mean(13,7)# connecting classes and functions to a Noden3=ComputeMean(n1.results,n2)graph.run()print(n3.results)# >>> 7.5Dask SupportZnFlow comes with support forDaskto run your graph:in parallel.through e.g. SLURM (seehttps://jobqueue.dask.org/en/latest/api.html).with a nice GUI to track progress.All you need to do is install ZnFlow with Daskpip install znflow[dask]. We
can then extend the example from above. This will runn1andn2in parallel.
You can investigate the graph on the Dask dashboard (typicallyhttp://127.0.0.1:8787/graphor via the client object in Jupyter.)importznflowimportdataclassesfromdask.distributedimportClient@znflow.nodifydefcompute_mean(x,y):return(x+y)/2@dataclasses.dataclassclassComputeMean(znflow.Node):x:floaty:floatresults:float=Nonedefrun(self):self.results=(self.x+self.y)/2withznflow.DiGraph()asgraph:n1=ComputeMean(2,8)n2=compute_mean(13,7)# connecting classes and functions to a Noden3=ComputeMean(n1.results,n2)client=Client()deployment=znflow.deployment.Deployment(graph=graph,client=client)deployment.submit_graph()n3=deployment.get_results(n3)print(n3)# >>> ComputeMean(x=5.0, y=10.0, results=7.5)We need to get the updated instance from the Dask worker viaDeployment.get_results. Due to the way Dask works, an inplace update is not
possible. To retrieve the full graph, you can useDeployment.get_results(graph.nodes)instead.Working with listsZnFlow supports some special features for working with lists. In the following
example we want tocombinetwo lists.importznflow@znflow.nodifydefarange(size:int)->list:returnlist(range(size))print(arange(2)+arange(3))>>>[0,1,0,1,2]withznflow.DiGraph()asgraph:lst=arange(2)+arange(3)graph.run()print(lst.result)>>>[0,1,0,1,2]This functionality is restricted to lists. There are some further features that
allow combiningdata: list[list]by either usingdata: list = znflow.combine(data)which has an optionalattribute=Noneargument to be used in the case of classes or you can simply usedata: list = sum(data, []).Attributes AccessInside thewith znflow.DiGraph()context manager, accessing class attributes
yieldsznflow.Connectorobjects. Sometimes, it may be required to obtain the
actual attribute value instead of aznflow.Connectorobject. It is not
recommended to run class methods inside thewith znflow.DiGraph()context
manager since it should be exclusively used for building the graph and not for
actual computation.In the case of properties or other descriptor-based attributes, it might be
necessary to access the actual attribute value. This can be achieved using theznflow.get_attributemethod, which supports all features fromgetattrand
can be imported as such:fromznflowimportget_attributeasgetattrHere's an example of how to useznflow.get_attribute:importznflowclassPOW2(znflow.Node):"""Compute the square of x."""x_factor:float=0.5results:float=None_x:float=None@propertydefx(self):returnself._x@x.setterdefx(self,value):# using "self._x = value * self.x_factor" inside "znflow.DiGraph()" would run# "value * Connector(self, "x_factor")" which is not possible (TypeError)# therefore we use znflow.get_attribute.self._x=value*znflow.get_attribute(self,"x_factor")defrun(self):self.results=self.x**2withznflow.DiGraph()asgraph:n1=POW2()n1.x=4.0graph.run()assertn1.results==4.0Instead, you can also use theznflow.disable_graphdecorator / context manager
to disable the graph for a specific block of code or theznflow.Propertyas a
drop-in replacement forproperty.GroupsIt is possible to create groups ofznflow.nodifyorznflow.Nodesindependent
from the graph structure. To create a group you can usewith graph.group(<name>). To access the group members, usegraph.get_group(<name>) -> list.importznflow@znflow.nodifydefcompute_mean(x,y):return(x+y)/2graph=znflow.DiGraph()withgraph.group("grp1"):n1=compute_mean(2,4)assertn1.uuidingraph.get_group("grp1")Supported FrameworksZnFlow includes tests to ensure compatibility with:"Plain classes"dataclassesZnInitattrsIt is currentlynotcompatible with pydantic. I don't know what pydantic
does internally and wasn't able to find a workaround.