Package: genpipes
GenpipesLibrary to write readable and reproductible pipelines using decorators and generators.
Tested for Python > 3.6.9.Installationpip install genpipesUsageBelow some use case on how to use the library.Quick StartThis quick start assume that you have pandas installed as dependency in you project.importpandasaspdfromgenpipesimportdeclare,composefromcollections.abcimportIterable@declare.generator()defdata_to_be_processed(path:str)->pd.DataFrame:df=pd.read_csv(path)returndf@declare.processor(inputs=["col1"])deffilter_by(stream:Iterable[pd.DataFrame],col_to_filter:str,value:str):fordfinstream:dff=df[df[col_to_filter]==value]yielddffpipe=compose.Pipeline(steps=[("fetching datasource from some csv file",data_to_be_processed,{}),("performing some filtering based on col1",filter_by,{"value":"some_value"})])output=pipe.run()Declaring a data sourceThe first task in data processing is usally to write code to acquire data. The library provide a decorator to declare your data source so they can be easily shared and readable.The decorators take in a list of inputs to be passed as positional arguments to the decorated function. This way you are binding arguments to the function but you are not hardcoding arguments inside the function.# my_datasource.pyimportpandasaspdfromgenpipesimportdeclare@declare.generator(inputs=["some_file.csv"])defdata_to_be_processed(path:str)->pd.DataFrame:df=pd.read_csv(path)returndf# other_file.pyfrommy_datasourceimportdata_to_be_processeddf=data_to_be_processed()However if you want to let some arguments be defined later you could use keywords arguments like so :# my_datasource.pyimportpandasaspdfromgenpipesimportdeclare@declare.generator(inputs=["some_file.csv"])defdata_to_be_processed(path:str,read_options:dict)->pd.DataFrame:df=pd.read_csv(path,**read_options)returndf# other_file.pyfrommy_datasourceimportdata_to_be_processeddf=data_to_be_processed(read_options={"encoding":"latin1"})Declaring generatorgeneratordecorator is use to initialize a stream.Function decorated are transformed to a Python generator object. You can decorate any function like a@generator.importpandasaspdfromgenpipesimportdeclare,compose@declare.generator(inputs=["some_file.csv"])defdata_to_be_processed(path:str)->pd.DataFrame:df=pd.read_csv(path)returndfOr a more complexe functionimportpandasaspdfromgenpipesimportdeclare,compose@declare.generator(inputs=["some_file.csv"])defdata_one(path:str)->pd.DataFrame:df=pd.read_csv(path)returndf@declare.generator(inputs=["some_file_bis.csv"])defdata_two(path:str)->pd.DataFrame:df=pd.read_csv(path)returndf@declare.generator(inputs=[data_one,data_two])defmerging_data(input_one:Callable,input_two:Callable)->pd.DataFrame:df_one=input_one()df_two=input_two()df_merged=df_one.merge(df_two,on="key")returndf_mergedDecorated function will not received the value from the stream. But the wrapper does receive the value from stream and push it downstream unchanged.That's why when calling  your function once decorated you have to pass it as first argument a generator object, so if you want to test you function you can do like that:empty_stream=()# use to feed the generator decorated functiongen=merging_data(empty_stream)df_merge=next(gen)# consumming merging_dataBecause the decorator returns a function that create a generator object you can create many generator object and feed several consumers.empty_stream=()# use to feed the generator decorated functiongen_one=merging_data(empty_stream)gen_two=merging_data(empty_stream)# multiple consumingconsumer_one=next(gen_one)consumer_two=next(gen_two)assertconsumer_one.equals(consumer_two)# TrueDeclaring processing functionsNow that we have seen how to declare data sources and how to generate a stream thanks to generator decorator. Let's see how to declare processing functions.importpandasaspdfromgenpipesimportdeclare,compose@declare.generator(inputs=["some_file.csv"])defdata_one(path:str)->pd.DataFrame:df=pd.read_csv(path)returndf@declare.generator(inputs=["some_file_bis.csv"])defdata_two(path:str)->pd.DataFrame:df=pd.read_csv(path)returndf@declare.generator(inputs=[data_one,data_two])defmerging_data(input_one:Callable,input_two:Callable)->pd.DataFrame:df_one=input_one()df_two=input_two()df_merged=df_one.merge(df_two,on="key")returndf_merged@declare.processor(inputs=[["col1, col2"]])defdeduplicate(stream:Iterable[pd.DataFrame],subset:List):fordfinstream:df_nodup=df[~df.duplicated(subset=[subset])]yielddf_nodupAs you can see,processordecorated functionMUST BEa generator function that take as first argument a generator that represent the stream of values.Composing pipelinesEven if we can use the decorator helper function alone, the library provide aPipelineclass that help to assemble functions decorated with bothgeneratorandprocessor.A pipeline object is compose of steps that aretuplewith 3 components:1- The description of the step2- The decorated function3- The keywords arguments to forward as dict, if none then empty dictimportpandasaspdfromgenpipesimportcompose,declare@declare.generator(inputs=["some_file.csv"])defdata_one(path:str)->pd.DataFrame:df=pd.read_csv(path)returndf@declare.generator(inputs=["some_file_bis.csv"])defdata_two(path:str)->pd.DataFrame:df=pd.read_csv(path)returndf@declare.generator(inputs=[data_one,data_two])defmerging_data(input_one:Callable,input_two:Callable)->pd.DataFrame:df_one=input_one()df_two=input_two()df_merged=df_one.merge(df_two,on="key")returndf_merged@declare.processor()defdeduplicate(stream:Iterable[pd.DataFrame],subset:List):fordfinstream:df_nodup=df[~df.duplicated(subset=[subset])]yielddf_noduppipe=compose.Pipeline(steps=[("data source is the merging of data one and data two",merging_data,{})# empty dict use here as there is no kwargs,("droping dups",deduplicate,{"subset":["col1"]})# forwarding subset as kwarg])When declaring pipeline objects we are not evaluating them. For that we need to call therunmethod. Therunmethod return the last object pulled out from the stream. In our case it will be the dedup dataframe from the last step.dedup_df=pipe.run()We can run the pipeline multiple time, it will re do all the steps:dedup_df=pipe.run()dedup_df_bis=pipe.run()assertdedup_df.equals(dedup_df_bis)# Truepipeline objects can be used in other pipeline instance as a step:@declare.processor()deffiltering_df(stream:Iterable[pd.DataFrame]):fordfinstream:dff=df.filter("some expr")yielddffother_pipe=compose.Pipeline(steps=[("take input other pipeline instance",pipe,{}),("filtering the output of the first pipe",filtering_df,{})])output_from_second_pipe=other_pipe.run()# will run the first pipe instance