Package: pipda
pipdaA framework for data piping in pythonInspired bysiuba,dfply,plydataanddplython, but with simple yet powerful APIs to mimic thedplyrandtidyrpackages in pythonAPI|Change Log|DocumentationInstallationpipinstall-UpipdaUsageVerbsA verb is pipeable (able to be called likedata >> verb(...))A verb is dispatchable by the type of its first argumentA verb evaluates other arguments using the first oneA verb is passing down the context if not specified in the argumentsimportpandasaspdfrompipdaimport(register_verb,register_func,register_operator,evaluate_expr,Operator,Symbolic,Context)f=Symbolic()df=pd.DataFrame({'x':[0,1,2,3],'y':['zero','one','two','three']})df#      x    y# 0    0    zero# 1    1    one# 2    2    two# 3    3    three@register_verb(pd.DataFrame)defhead(data,n=5):returndata.head(n)df>>head(2)#      x    y# 0    0    zero# 1    1    one@register_verb(pd.DataFrame,context=Context.EVAL)defmutate(data,**kwargs):data=data.copy()forkey,valinkwargs.items():data[key]=valreturndatadf>>mutate(z=1)#    x      y  z# 0  0   zero  1# 1  1    one  1# 2  2    two  1# 3  3  three  1df>>mutate(z=f.x)#    x      y  z# 0  0   zero  0# 1  1    one  1# 2  2    two  2# 3  3  three  3Functions used as verb arguments# verb can be used as an argument passed to another verb# dep=True make `data` argument invisible while calling@register_verb(pd.DataFrame,context=Context.EVAL,dep=True)defif_else(data,cond,true,false):cond.loc[cond.isin([True]),]=truecond.loc[cond.isin([False]),]=falsereturncond# The function is then also a singledispatch generic functiondf>>mutate(z=if_else(f.x>1,20,10))#    x      y   z# 0  0   zero  10# 1  1    one  10# 2  2    two  20# 3  3  three  20# function without data argument@register_funcdeflength(strings):return[len(s)forsinstrings]df>>mutate(z=length(f.y))#    x     y    z# 0  0  zero    4# 1  1   one    3# 2  2   two    3# 3  3 three    5ContextThe context defines how a reference (f.A,f['A'],f.A.Bis evaluated)@register_verb(pd.DataFrame,context=Context.SELECT)defselect(df,*columns):returndf[list(columns)]df>>select(f.x,f.y)#    x     y# 0  0  zero# 1  1   one# 2  2   two# 3  3 threeHow it worksdata%>%verb(arg1,...,key1=kwarg1,...)The above is a typicaldplyr/tidyrdata piping syntax.The counterpart python syntax we expect is:data>>verb(arg1,...,key1=kwarg1,...)To implement that, we need to defer the execution of theverbby turning it into aVerbobject, which holds all information of the function to be executed later. TheVerbobject won't be executed until thedatais piped in. It all thanks to theexecutingpackage to let us determine the ast nodes where the function is called. So that we are able to determine whether the function is called in a piping mode.If an argument is referring to a column of the data and the column will be involved in the later computation, the it also needs to be deferred. For example, withdplyrinR:data%>%mutate(z=a)is trying add a column namedzwith the data from columna.In python, we want to do the same with:data>>mutate(z=f.a)wheref.ais aReferenceobject that carries the column information without fetching the data while python sees it immmediately.Here the trick isf. Like other packages, we introduced theSymbolicobject, which will connect the parts in the argument and make the whole argument anExpressionobject. This object is holding the execution information, which we could use later when the piping is detected.Documentationhttps://pwwang.github.io/pipda/See alsodatarfor real-case usages.