Package: lwpipe
Overviewlwpipeprovides a lightweight pipeline. lwpipe has fewer features thanluigiorKedro, but you can quickly build and run pipelines.Note that lwpipe is highly inspired byKedro.Installationfrom pypi:pipinstalllwpipeUsageMinimal example (of course, no need to use this library..):fromlwpipeimportNode,Pipelinenodes=[Node(func=lambdax,y:x+y,inputs=(1,2)),Node(func=lambdax:x**2),]pipe=Pipeline(nodes)outputs=pipe.run()assertoutputs[0]==9Example with interim data output:fromlwpipeimportNode,Pipelinefromlwpipe.ioimportdump_pickle,load_pickledeftime_consuming_func(x):returnxnodes=[Node(func=time_consuming_func,inputs=100,outputs_dumper=dump_pickle,outputs_path="interim_data.pickle",outputs_loader=load_pickle,# needed to bypass this node),Node(func=lambdax:x**2,name="square"),]pipe=Pipeline(nodes)outputs=pipe.run()assertoutputs[0]==10000Once the first node is executed, you can bypass the node bypipe.run(1)orpipe.run("square").Multiple outputs with numpy:importnumpyasnpfromlwpipeimportNode,Pipelinefromlwpipe.ioimportdump_npy,load_npydefsplit(x):returnx[:5],x[5:]nodes=[Node(func=split,inputs=np.arange(10),outputs=("former","latter"),outputs_dumper=dump_npy,outputs_path=("df1.npy","df2.npy"),outputs_loader=load_npy,),Node(func=np.mean,name="former_mean",inputs="former",# calculated at the first nodeoutputs="former_mean",),Node(func=np.mean,name="latter_mean",inputs="latter",# calculated at the first nodeoutputs="latter_mean",),]pipe=Pipeline(nodes)outputs=pipe.run()assertoutputs[0]==7.0# You can access interim results by "results" dictassertpipe.results["former_mean"]==2.0batch dump example (return values are dumped to one file):importnumpyasnpfromlwpipeimportDumpType,Node,Pipelinefromlwpipe.ioimport(dump_dict_pickle,dump_savez_compressed,load_dict_pickle,load_savez_compressed,)defdivide(x):returnx[:,0],x[:,1]nodes=[Node(func=divide,inputs=np.arange(1,7).reshape((3,2)),outputs=("mean_a","mean_b"),outputs_dumper=dump_dict_pickle,outputs_dumper_type=DumpType.BATCH,outputs_path="1.pickle",outputs_loader=load_dict_pickle,),Node(func=lambdax,y:(x,y),outputs=("a","b"),outputs_dumper=dump_dict_pickle,outputs_dumper_type=DumpType.BATCH,outputs_path="2.pickle",outputs_loader=load_dict_pickle,),Node(func=lambdax,y:(x.max(),y.max()),inputs=("a","b"),outputs=("c","d"),outputs_dumper=dump_savez_compressed,outputs_dumper_type=DumpType.BATCH,outputs_path="3.npz",outputs_loader=load_savez_compressed,)]pipe=Pipeline(nodes)outputs=pipe.run()assertoutputs==(5,6)you can pass a config object to a function:fromlwpipeimportNode,Pipelinedefadd(a,cfg):returna+cfg["hyperparam"]nodes=[Node(func=add,inputs=5,config={"hyperparam":10})]# equivalent to# nodes = [Node(func=lambda a: add(a, {"hyperparam": 10}), inputs=5)]pipe=Pipeline(nodes)outputs=pipe.run()assertoutputs[0]==15defdumper(data,filepath,cfg):filepath=Path(filepath)filepath=filepath.with_name(filepath.name+cfg["hyperparam"])returndump_pickle(data,filepath)# also, outputs_dumper can take config as its argumentnodes=[Node(func=add,inputs=5,config={"hyperparam":10},outputs_dumper=dumper,outputs_dumper_take_config=True)]Pipelinealso accepts a list of functions with no arguments and return-values:fromlwpipeimportPipelinedeffunc():returnfuncs=[func,func]pipe=Pipeline(funcs)pipe.run()# equivalent to# for func in funcs:#   func()# you can specify names of functionspipe=Pipeline(funcs,names=["func1","func2"])pipe.run()More examples are included in thetest cases.