Package: pipedown
Pipedown :shushing_face:Pipedown is a machine learning model pipelining package for Python.  It doesn't
reallydomuch, it just lets you define a directed acyclic graph (DAG) of
modeling steps, each of which is defined as a class withfitandrunmethods (likescikit-learn), and it runs your nodes
in order and provides a visualization of the DAG.Pipedown is designed around:Asingle code path: use the same code for training, validation, and inference on new test data.Modularityandtestability: each node is defined as its own class withfit()andrun()methods, making it easy to unit test each node.Visibility: pipedown comes with an html viewer to explore the structure of your DAGs, and shows links to the source code associated with each node.Portability: pipedown models can easily be trained in one environment (e.g. a batch job), serialized, and then loaded into another environment (e.g. a model server) for inference.State: DAG nodes can store state; they aren't just stateless functions.Flexibility: pipedown allows you to define models as DAGs instead of just linear pipelines (likescikit-learn), but doesn't force your project to have a specific file structure (likeKedro).Pipedown is NOT an ETL / data engineering / task scheduler tool - for that use
something like Airflow, Argo, Dask, Prefect, etc.  You can do some basic and
inefficient data processing with Pipedown, but really it's focused on creating
portable model pipelines.Git repository:http://github.com/brendanhasz/pipedownDocumentation:Bug reports:http://github.com/brendanhasz/pipedown/issuesRequirementsTo use the visualization tools, you need to havegraphvizinstalled.  On Ubuntu, you can install with:sudoapt-getinstallgraphvizInstallationJust use pip!pipinstallpipedownGetting StartedYou can define nodes in your pipeline by creating a class which inherits fromNode.  Your nodes must define arun()method, which performs some
transformation on its input data, and can also optionally define afit()method to store some state.  For example, a simple node which performs mean
imputation would look like:importnumpyasnpimportpandasaspdimportpipedownclassMeanImputer(pipedown.nodes.base.Node):deffit(self,X:pd.DataFrame,y:pd.Series):self.means=X.mean()defrun(self,X:pd.DataFrame,y:pd.Series):forcinX:X[X[c].isnull()]=self.means[c]returnX,yOr, as another example, a node to load some data from a CSV file:classLoadFromCsv(pipedown.nodes.base.Node):def__init__(self,filename:str):self.filename=filenamedefrun(self):returnpd.read_csv(self.filename)Similarly, you can define models by creating aModelclass withfit()andpredict()methods:classLinearRegression(pipedown.nodes.base.Model):deffit(self,X:pd.DataFrame,y:pd.Series):xx=X.valuesyy=y.values.reshape((-1,1))self.weights=np.linalg.inv(xx.T@xx)@xx.T@yy# least squaresdefpredict(self,X:pd.DataFrame):y_pred=X.values@self.weightsreturnpd.Series(data=y_pred.ravel(),index=X.index)And finally, you can assemble your nodes into a directed acyclic graph (DAG) by
creating a class which inherits fromDAG.  That class should implement anodes()method, which returns a dictionary of all the nodes in your DAG (keys
are node names, and values are the node objects), and also anedges()method,
which returns a dictionary mapping nodes to their parents.Pipedown also comes with a few built-in nodes which you can use in your DAG,
for example:Input- to represent test data inputPrimary- to split the data into features and the target variableMetrics, to evaluate your model's performancefrompipedown.dagimportDAGfrompipedown.nodes.baseimportInput,Primaryfrompipedown.nodes.metricsimportMeanSquaredErrorclassMyModel(pipedown.dag.DAG):defnodes(self):return{"load_csv":LoadFromCsv("some_csv.csv"),"test_input":Input(),"primary":Primary(["feature1","feature2"],"target"),"imputer":MeanImputer(),"lr":LinearRegression(),"mse":MeanSquaredError(),}defedges(self):return{"mse":"lr","lr":"imputer","imputer":"primary","primary":{"test":"test_input","train":"load_csv"},}Then, you can instantiate your DAG and fit the whole thing:model=MyModel()model.fit()Run it on new test data:>>>test_input={"feature1":1.2,"feature2":3.4}>>>model.run(inputs={"test_input":test_input},outputs="lr")3.14159Get cross-validated predictions:cv_predictions=model.cv_predict()Or cross-validated metrics:model.cv_metric()model_namemetric_namefoldmetric_valuelrmean_squared_error01.023lrmean_squared_error11.154lrmean_squared_error20.935lrmean_squared_error31.215lrmean_squared_error41.103You can serialize/deserialize the whole model usingcloudpickle.model.save("my_model.pkl")loaded_model=pipedown.dag.io.load_dag("my_model.pkl")And you can get an html viewer for visualizing the DAG!# Get the raw htmlraw_html=model.get_html()# Or, save to html file:model.save_html("my_model.html")ThanksSyntax higlighting in the DAG viewer is done usinghiglight.jswith the great Atom One Dark ReasonML theme by Gidi Morris / Daniel Gamage.