Package: pyflow-viz
PyflowPyflow is a light weight library that lets the user construct a directed acyclic computation graph (DAG) that evaluates lazily. It can cache intermediate results, only compute the parts of the graph that has data dependency, and immediately release memory of data whose dependecy is no longer required. Pyflow is simple and light, built purely on Python, using the weak references for memory management and doubly linked list for DAG construction.Unlike computation graph based engines such as Dask or PySpark, Pyflow is not meant to be a parallel data processor, or to change the way computation resources are used. Instead, it is meant to be a light weight tool for code organization in the form of DAG and for graph visualization that can be used on top of Dask or PySpark.Installpip install pyflow-vizGetting startedLet’s construct a simple computation graph: (Note the similarity of API to that of Keras functional API!)frompyflowimportGraphBuilderdefadding(a,b):returna+bG=GraphBuilder()a1=G.add(adding)(2,2)# you add methods with `add` instance method.a2=G.add(adding)(3,a1)a3=G.add(adding)(a1,a2)At this point, no evaluation has occurred. The outputsa1,a2, anda3areDataNodeobjects. The methods that we just added areOperationNodeobjects of the DAG.You can easily visualize the resulting DAG usingviewmethod:G.view()The default setting of theviewmethod will only visualize the operation nodes. Butviewcan do much more, as we will learn shortly.You can execute the computation graph by invoking therunmethod:G.run()# will run all the operation nodesYou can pass in data nodes to get the desired results back this way:a1_result,a3_result=G.run(a1,a3)# will run all the operation nodes, and return the result data values of a1, a3But what if you don’t want to run every method in the DAG? There isrun_onlymethod for that, which we will learn shortly.A couple notes:The API was inspired by that of theKeras functional APIFor demo, we are using a simple method of adding two integers, but the input method can be any python function, including instance methods, with arbitrary inputs such as numpy array, pandas dataframe or Spark dataframe.Multi-output methodsWhat if we have a python function with multiple outputs? Due to dynamic nature of python, it is impossible to determine the number of outputs before the function is actually ran. In such a case, youmustspecify the number of outputs byn_outargument. Otherwise, Pyflow will deem the output to be a single output whose value is a list of multiple elements. Here is an example of how to do usen_outparameter to create multiple child output nodes:frompyflowimportGraphBuilderdefadding(a,b):returna+bdefmulti_output_method(a,b):returna+1,b+1G=GraphBuilder()a1=G.add(adding)(2,2)a2,b2=G.add(multi_output_method,n_out=2)(a1,2)# n_out argument!a3=G.add(adding)(a2,3)a4=G.add(adding)(b2,a1)G.view()Visualizing data flowTheviewfunction actually has the ability to summarize the DAG by only showing the user theOperationNodes, which it does by default. We can override this default setting by using thesummaryparameter of the function:G.view(summary=False)With the summary functionality turned off, the complete DAG visualization will includesDataNodesas well as theOperationNodes. You may be wondering what the extra records with(1, )written inside are. They signal the data persistence. We will discuss what this is, and how this works, in greater detail later.But that graph image is a little too big. We can shrink the gap between the nodes with handy thegapparameter:G.view(summary=False,gap=0.2)# the default value is 0.415Styling your DAGPyflow lets the user customize the DAG visuals to a certain degree, with more to come in the future. Let’s take a look at some examples.frompyflowimportGraphBuilderdefquery_dataframe_A():return1# pretend this was a pandas or Spark dataframe!defquery_dataframe_B():return2defproduct_transform(inp):returninp*2defjoin_transform(inp1,inp2):returninp1+inp2defsplit_transform(inp):returninp+1,inp+2G=GraphBuilder()df1=G.add(query_dataframe_A)()df2=G.add(query_dataframe_B)()new_df1=G.add(product_transform)(df1)new_df2=G.add(product_transform)(df2)dfa,dfb=G.add(split_transform,n_out=2)(new_df2)joined_df=G.add(join_transform)(new_df1,dfa)G.view()But since at a conceptual level, queries are similarly progenitors of new data, perhaps we want to put them side by side on top, and position is controlled byrankparameter. Also, since these are probably coming from some data storage, we might want to style their nodes accordingly, with different color.G=GraphBuilder()df1=G.add(query_dataframe_A,rank=0,shape='cylinder',color='lightblue')()df2=G.add(query_dataframe_B,rank=0,shape='cylinder',color='lightblue')()new_df1=G.add(product_transform)(df1)new_df2=G.add(product_transform)(df2)dfa,dfb=G.add(split_transform,n_out=2)(new_df2)joined_df=G.add(join_transform)(new_df1,dfa)G.view()But then we might want to make the DAG a little shorter, especially if we are to add more and more intermediate steps. We can control more detailed aesthetics withgraph_attributes(thegapis simply the short cut parameter for this!):graph_attributes={'graph_ranksep':0.25}G.view(graph_attributes=graph_attributes)You can take a look and play around with the rest of the configurations:G.graph_attributes# the default settings are found at:G.default_graph_attributes# 'data_node_fontsize': 10,# 'data_node_shape': 'box',# 'data_node_color': None,# 'op_node_fontsize': 12,# 'op_node_shape': 'box',# 'op_node_color': 'white',# 'graph_ranksep': 0.475,# 'graph_node_fontsize': 12.85,# 'graph_node_shape': 'box3d',# 'graph_node_color': 'white',# 'graph_node_shapesize': 0.574,# 'persist_record_shape': TrueFinally, you can set the alias of the nodes by passing inmethod_aliasand/oroutput_aliasin theaddmethod. Themethod_aliaswill set the alias of the operation node being added, andoutput_aliaswill set the alias of the child data node of that operation node.G=GraphBuilder()dfa=G.add(query_dataframe_A,rank=0,shape='cylinder',color='lightblue',output_alias='df_A')()dfb=G.add(query_dataframe_B,rank=0,shape='cylinder',color='lightblue',output_alias='df_B')()dfa1=G.add(product_transform)(dfa)dfb1=G.add(product_transform)(dfb)# note the list of alias for n_out = 2dfa,dfb=G.add(split_transform,n_out=2,output_alias=['first_out','second_out'])(dfa1)joined_df=G.add(join_transform,output_alias='final_data')(dfb1,dfa)graph_attributes={'graph_ranksep':0.25}G.view(summary=False,graph_attributes=graph_attributes)The default alias for operation node is the String name of the method being passed in, and the default alias for data node is simply “data”. We do not include the example of settingmethod_aliasto discourage its use. Setting method alias different from the method name will make look up of graph node in the code base very difficult.No output methodsOften when we are processing data, we will end up doing something with that data, whether it is to upload it somewhere, save it somewhere, or use pass it to a model, etc. In those cases, we do not expect any return data.# this method does not have return statementdefsave_data(data):# save the data somewhere# no return statement neededpassPyflow will create graph accordingly, such that the outputless operation node is a leaf node.frompyflowimportGraphBuilderdefquery_dataframe_A():return1# pretend this was a pandas or Spark dataframe!defquery_dataframe_B():return2defproduct_transform(inp):returninp*2defjoin_transform(inp1,inp2):returninp1+inp2defsplit_transform(inp):returninp+1,inp+2defsave_data(data):# save the data somewhere# no return statement neededpassG=GraphBuilder()df1=G.add(query_dataframe_A,rank=0,shape='cylinder',color='lightblue')()df2=G.add(query_dataframe_B,rank=0,shape='cylinder',color='lightblue')()new_df1=G.add(product_transform)(df1)new_df2=G.add(product_transform)(df2)dfa,dfb=G.add(split_transform,n_out=2)(new_df2)joined_df=G.add(join_transform)(new_df1,dfa)G.add(save_data)(dfb)G.add(save_data)(joined_df)graph_attributes={'graph_ranksep':0.25}G.view(summary=False,graph_attributes=graph_attributes)This is a more realistic shape of the DAG in the actual use case of data preprocessing.Executing parts of graphTherunmethod will execute all nodes in the graph, but what if you don’t want to run every node in the graph to save yourself time? Let’s look at an example:frompyflowimportGraphBuilderdefquery_dataA():return1defquery_dataB():return2defquery_dataC():return3deftransform_dataA(a):returnadeftransform_dataB(a):returnadeftransform_dataC(a):returnadefjoin_dataAB(a,b):returna+bdefsave_dataAB(ab):passdefjoin_dataC(a,c):returna+cG=GraphBuilder()a=G.add(query_dataA,rank=0)()b=G.add(query_dataB,rank=0)()c=G.add(query_dataC,rank=0)()a=G.add(transform_dataA)(a)b=G.add(transform_dataB)(b)c=G.add(transform_dataC)(c)ab=G.add(join_dataAB)(a,b)G.add(save_dataAB)(ab)abc=G.add(join_dataC)(ab,c)G.view(gap=0.25)From the above graph, let’s say you want to test thetransform_dataAmethod. For this purpose, you only need to runquery_dataAandtransform_dataA. In such a case, you can use therun_onlymethod, instead ofrunmethod, which will execute every node in the graph:a_result=G.run_only(a)# 1When you invoke therun_onlymethod, Pyflow will only execute parts of the graph that has the data dependency to the asked node.This time, let’s say you are testing thesave_dataABmethod. But this node does not have data node that we can use to pass into therun_onlymethod. That’s why you can also pass in the string of the node names into therun_onlymethod:a_result=G.run_only(a,'save_dataAB')In the above code, only the result foranode is returned becausesave_dataABdoes not have a return statement.Visualizing computation dependencyPyflow will only execute parts of the graph that has data dependency. We can visualize this dependency withview_dependencymethod. We will use the same graph from the previous example:G.view_dependency('save_dataAB')You can pass in several arguments, just as you can withrun_onlymethod for execution:G.view_dependency('save_dataAB',c)This method also supports other parameters ofviewmethod:G.view_dependency('save_dataAB',c,summary=False,gap=0.2)Grafting graphs togetherWhen the computation graph becomes too big, the size of the visualized graph can actually end up becoming a hinderance to clean data flow documentation. Not only that, we could also benefit at the conceptual code organization level, if we had the ability to define multiple graphs and combine them together flexibly. I.e. we could treat a graph as if it was just another operation node. As of version0.7, we can do this. Let’s look at an example:frompyflowimportGraphBuilderdefadding(a,b):returna+bG=GraphBuilder(alias='First Graph')# notice alias at graph level!a1=G.add(adding)(1,2)a2=G.add(adding)(a1,2)a3=G.add(adding,output_alias='leaving!')(a1,a2)G.view()Let’s look at the unsummarized version to take notice of the output_alias of the last data node:# let's make it a little shorter with ranksep parameter we talked about earlier!G.view(summary=False,graph_attributes={'graph_ranksep':0.3})In the above code, we have created one graph. But we can create another graph, and graft theFirst Graphgraph to the new graph:H=GraphBuilder(alias='Second Graph')b1=H.add(adding)(1,3)b2=H.add(adding)(b1,a3)b3=H.add(adding)(b1,b2)H.view(summary=False)# notice that the output_alias from previous graph is also preserved!As you can see, the previous graph is now summarized into a box. You can combine as many graphs in this way as you want. Despite this visual effect,b3is now part of one single big combined computation graph. Therefore, callingb3.get()will trigger computations in nodes that belong to bothGandHas long as they are needed. As far as computation is concerned, you just have one big graph.Saving your DAG imageYou can easily save your DAG image by invokingsave_viewmethod, which returns the file path of the saved image:G.save_view()Thesave_viewmethod also hassummaryboolean parameter. You can also set the file name and file path by passing indirpathandfilenameparameter. They default to current working directory and “digraph” respectively. You can also set the file format as png or pdf by settingfileformatparameter. The default is png.HTML documentation of DAGWith the visualization of the DAG, we can see the input-output relations among the functions, but it alone does not tell what each of the function does. But you can create a single HTML documentation that tells the complete semantic story of the DAG, usingdocumentmethod:frompyflowimportdocumentdocument(G)# or document(G, H, I) etc if you have more than one graphDoing so will create a static HTML file that displays the DAG image as well as the docstrings of each of the functions that goes into the DAG on the right side, which you can scroll through.frompyflowimportGraphBuilderfrompyflowimportdocumentdefmethodA(elem):"""Some descriptions of the methodA

        Parameter
        ---------
        elem : int
        """returnelemdefmethodB(elem):"""Some descriptions of the methodB

        Parameter
        ---------
        elem : int
        """returnelemdefmethodC(elem):"""Some descriptions of the methodC

        Parameter
        ---------
        elem : int
        """returnelemG=GraphBuilder()a=G.add(methodA)(3)b=G.add(methodB)(a)c=G.add(methodC)(b)document(G)This code will produce the following HTML file:Memory persistance with PyflowYou have the option of either persisting all of the intermediate results, or persisting part of the intermediate results.To persist all intermediate results, usepersistparameter atGraphBuilderlevel:frompyflowimportGraphBuilderG=GraphBuilder(persist=True)# set persist to Truea1=G.add(adding)(1,2)a2,a3=G.add(return2,n_out=2)(a1,3)a4=G.add(adding)(a1,5)a5=G.add(adding)(a4,a3)a5.get()With persist enabled, after runninga5.get(), when you try to runa4.get(), the graph will not recompute anything becausea4node result will have been cached in memory. The persist is turned off by default, as it is assumed that the user of the pyflow will process large amounts of data.To persist parts of the data, you can specify thepersistparameter ataddlevel:frompyflowimportGraphBuilderG=GraphBuilder(persist=False)# default valuea1=G.add(adding)(1,2)a2,a3=G.add(return2,n_out=2)(a1,3)a4=G.add(adding,persist=True)(a1,5)# persist herea5=G.add(adding)(a4,a3)a5.get()Then, when you runa4.get()it will not rerun the computation asa4result has been cached in memory although all other intermediate results will have been released.At last, we can understand the difference betweenrun()andrun(a1, a3). Even if you don’t persist anything, either at the graph level or the node level, by passing in thea1, a3, the graph will automatically persist their data for you, and return the persisted data by internally invokingget()on the nodesa1, a3. The rest of data nodes are subject to the same immediate memory release mechanism.In terms of the codes, these two are equivalent:# run() with arguments:frompyflowimportGraphBuilderG=GraphBuilder()a1=G.add(adding)(1,2)a2=G.add(adding)(a1,3)a3=G.add(adding)(a2,a1)a1_val,a3_val=G.run(a1,a3)# run() without arguments:G=GraphBuilder()a1=G.add(adding,persist=True)(1,2)a2=G.add(adding)(a1,3)a3=G.add(adding)(a2,a1)G.run()a1_val=a1.get()a3_val=a3.get()Also, when you persist certain nodes, this persistence request will manifest in the graph by an empty record box:frompyflowimportGraphBuilderG=GraphBuilder()a1=G.add(adding)(1,2)a2=G.add(adding)(a1,3)a3=G.add(adding)(a2,a1)G.view()The empty box signifies that the graph is requested to persist that data, but it does not yet hold that data because it has not yet been executed. But once you run the graph, the empty record slot will be filled by the dimensionality of the resulting data. Currently it supports PySpark dataframe, numpy array, and pandas dataframe. All other data will have a default dimension of(1, ).G.run()Now, of course, it is not the method that is being persisted but the resulting data of that op node. You can see this when you visualize the DAG withsummary=False:G.run(summary=False)Some notes:The op node with record box is a short hand way of conveying the message that the child data node of that op node will be persisted.The raw data are automatically persisted, which is why you see the dimensionality information in the record box. This is because the raw user data inputs cannot be recomputed from the graph alone. But this will not be visible whensummary=True, because the op node will only show the record box for persisted child data node, and user supplied inputs will always be parent data node.Although this is not made explicitly visible, the final leaf data node are always persisted whenrunmethod is invoked. But this will not be explicitly shown in the graph unless the user manually suppliespersistflag at theaddmethod invocation.Lastly, thepersistflag is interoperable with Spark when PySpark dataframe is the data type. This means, when you persist the data using the DAG, if the underlying data is a PySpark dataframe, the Pyflow will persist the dataframe for you. However, unpersisting is not done by the Pyflow. If you want to unpersist a dataframe, do so manually.Computation and memory efficiency of Pyflow (OUTDATED)When you invokegetmethod, pyflow will only then evaluate, and it will evaluate only the parts of the graph that is needed to be evaluated. Also, as soon as an intermediate result has no dependency, it will automatically release the memory back to the operating system. Let’s take a tour of the computation process to better understand this mechanism by turning onverboseparameter.frompyflowimportGraphBuilderdefadding(a,b):returna+bdefmulti_output_method(a,b):returna+1,b+1G=GraphBuilder(verbose=True)# set verbose to Truea1=G.add(adding)(1,2)a2,a3=G.add(return2,n_out=2)(a1,3)a4=G.add(adding)(a1,5)a5=G.add(adding)(a4,a3)a5.get()Withverbose=True, along with the final output, pyflow will also produce the following standard output:computing for data_12
adding_11 activated!
adding_8 activated!
adding_0 activated!
return2_4 activated!
computing for data_10
computing for data_3
running adding_0
adding_0 deactivated!
running adding_8
data_3 still needed at return2_4
adding_8 deactivated!
computing for data_7
running return2_4
data_3 released!
return2_4 deactivated!
running adding_11
data_10 released!
data_7 released!
adding_11 deactivated!Let’s take the tour of this process by looking at the graph. Notice that in verbose mode, the graph will actually print out the uid’s of the nodes not just their aliases (more on setting alias later!)As pyflow tries to computedata_12, it will first activate all theOperationNodesthat is needed for the computation, in our case, those areadding_11,adding_8,adding_0,return2_4. It will then follow the lineage of the graph to work on intermediate results needed to proceed down the graph. Notice that as the computation proceeds, theOperationNodesthat were activated are deactivated. When it gets todata_3, notice that it is needed at bothadding_8andreturn2_4. Thus, once it completesadding_8, it cannot yet release the memory fromdata_3:data_3 still needed at return2_4. But as soon asreturn2_4is ran, it releasesdata_3from memory, as it is not needed anymore:data_3 released!. TheDataNodeswith raw inputs such as integers are not released since there is no way for the graph to reconstruct them.By the same token, if you were to run the graph from middle, say, ata4:a4.get()You will see:computing for data_10
adding_8 activated!
adding_0 activated!
computing for data_3
running adding_0
adding_0 deactivated!
running adding_8
data_3 released!
adding_8 deactivated!In this case, sincereturn2_4is not activated, thedata_3does not consider its presence in deciding release of memory.On the other hand,runmethod will activatealloperation nodes. This will make sure that even the operation nodes that do not have children are ran. However, the immediate memory release mechanism still applies torunmethod, unless otherwise specified. Refer below.