Package: conduits
Conduits - A Declarative Pipelining Tool For PandasTraditional tools for declaring pipelines in Python suck. They are mostly
imperative, and can sometimes requires that you adhere to strong contracts in
order to use them (looking at you Scikit Learn pipelines ಠ_ಠ). It is also
usually done completely differently to the way the pipelines where developed
during the ideation phase, requiring significate rewrite to get them to work
in the new paradigm.Modelled off the declarative pipeline of Flask,Conduitsaims to give you a
nicer, simpler, and more flexible way of declaring your data processing pipelines.InstallationpipinstallconduitsQuickstartimportpandasaspdfromconduitsimportPipeline############################ Pipeline Declaration ############################pipeline=Pipeline()pipeline["transformed"]=False@pipeline.step(dependencies=["first_step"])defsecond_step(data,*,adder=0):returndata+adder@pipeline.step()deffirst_step(data,*,power=1):returndata**power@pipeline.step(dependencies=["second_step"])defthird_step(data,fit:bool,transform:bool):iftransform:pipeline["transformed"]=Truereturndata################# Execution #################df=pd.DataFrame({"X":[1,2,3],"Y":[10,20,30]})assertpipeline["transformed"]==Falseoutput=pipeline.fit_transform(df,adder=1,power=2)assertoutput.X.sum()!=29# Addition before square => False!assertoutput.X.sum()==17# Square before addition => True!assertpipeline["transformed"]==Truepipeline.save("pipeline.joblib")reloaded=Pipeline().load("pipeline.joblib")assertreloaded["transformed"]==True# State is persisted on reload.Usage GuideDeclarationsPipeline DecoratorYour pipeline is defined using a standard decorator syntax. You can wrap your
pipeline steps using the decorator:@pipeline.step()deftransformer(df):returndf+1Function SignatureThe decorated function's signature carries all the information Conduits needs
to infer what functionality it needs to activate. You can pass in a single
dataframe or many dataframes into the signature, and this will be carried
through during thefit(),transform(), andfit_transform()calls. Hence
both function signatures would work:pipeline=Pipeline()...@pipeline.step()defsingle_frame(data):returndata+1...df=pipeline.fit_transform(df)pipeline=Pipeline()...@pipeline.step()defXy_transfomer(X,y):returnX+1,y...X,y=pipeline.fit_transform(X,y)You can also define hyperparameters that your function has access to by including
them in the function signature after the*separator (seePEP 3102). It is recommended that
you set a default value for the hyperparameter but it is not necessarypipeline=Pipeline()...@pipeline.step()defadder(data,*,n):returndata+n@pipeline.step()defmultiplier(data,*,m=1):returndata*m...df=pipeline.fit_transform(df)# Fail! `n` isn't passed indf=pipeline.fit_transform(df,n=1)# Will succeed! `n`=1, `m`=1df=pipeline.fit_transform(df,n=1,m=2)# Will succeed! `n`=`, `m`=2Stateful TransformersIf your transformer is stateful, you can optionally supply the function withfitandtransformboolean arguments. They will be set asTruewhen the
appropriate method is called. Both arguments are optional and independent of
one another (i.e. you can just have thefitargument without thetransformargument).@pipeline.step()defstateful(data:pd.DataFrame,fit:bool,transform:bool):iffit:scaler=StandardScaler()pipeline["scaler"]=scaler.fit(data)iftransform:data=pipeline["scaler"].transform(data)returndataPipeline SerialisationYou should not serialise the pipeline object itself. Rather, you should
use thepipeline.save(path)andpipeline.load(path)to handle serialisation
and deserialisation.If there are any dependencies between your pipeline steps, you may specify these
in your decorator and they will be run prior to this step being run in the
pipeline. If a step has no dependencies specified it will be assumed that it can
be run at any point.@pipeline.step(dependencies=["add_feature_X","add_feature_Y"])defcombine_X_with_Y(df):returndf.X+df.YAPIConduits attempts to mock the Scikit Learn API as best as possible. Your defined
piplines have the standard methods of:pipeline.fit(df)out=pipeline.transform(df)out=pipeline.fit_transform(df)Note that for the current release you can only supply pandas dataframes or
series objects. It will not accept numpy arrays.You can save artifacts into the pipeline using standard dictionary notation.pipeline["artifact"]=[1,2,3]artifact=pipeline["artifact"]You can serialise all artifacts within the pipeline using thepipeline.save(path)andpipeline.load(path)methods. The pipeline will be serialised using the
joblib library.pipeline=Pipeline()...pipeline.save("pipeline.joblib")pipeline=Pipeline().load("pipeline.joblib")TestsIn order to run the testing suite you should install thedev.requirements.txtfile. It comes with all the core dependencies used in testing and packaging.
Once you have your dependencies installed, you can run the tests via the target:maketestsThe tests rely onpytest-regressionsto test some functionality. If you make
a change you can refresh the regression targets with:makeregressions