Package: pipecutter
pipecutterpipecutter provides a few tools for luigi such that it works better with data science libraries and environments such as pandas, scikit-learn, and Jupyter notebooks.Table of contentsInstallationUsageDebug in an interactive environmentTargetsFull exampleInstallationpipinstallpipecutterPython 3.6+ is required. pipecutter followssemantic versioning.Usagepipecutter currently providesa more convenient way to run and debug luigi tasks in interactive environments such as Jupyter notebookssome luigi targets for saving pandas dataframes to parquet, scikit-learn models with joblib, ...Debug in an interactive environmentWith luigi, you can already run tasks in a Python script/Jupyter notebook/Python console by using theluigi.buildfunction (probably withlocal_scheduler=Trueas arugment). However, if the tasks throws an exception this will be caught by luigi and you are not able to drop into a post mortem debugging session.pipecutter.runis a light wrapper aroundluigi.buildwhich disables this exception handling.In[1]:importluigiIn[2]:importpipecutterIn[3]:classTaskWhichFails(luigi.Task):...:defrun(self):...:raiseException("Something is wrong")# Traceback below is shortened for readabilityIn[4]:pipecutter.run(TaskWhichFails())---------------------------------------------------------------------------ExceptionTraceback(mostrecentcalllast)<ipython-input-5-a970d52d810a>in<module>---->1pipecutter.run(TaskWhichFails())...<ipython-input-3-4e27674090fa>inrun(self)1classTaskWhichFails(luigi.Task):2defrun(self):---->3raiseExceptionException:Somethingiswrong# Drop straight into the debuggerIn[5]:%debug><ipython-input-6-e7528a27d82e>(3)run()1classTaskWhichFails(luigi.Task):2defrun(self):---->3raiseException4ipdb>This should reduce the barrier for already using luigi tasks while developing a model and thereby making it easier to move into production later on.Additionally, you can print the dependencies of tasks withpipecutter.print_tree(wrapper aroundluigi.tools.deps_tree.print_tree) or build a graphviz Graph withpipecutter.build_graphwhich you can save as .png, .pdf, etc. or directly view in your Jupyter notebook. See the Full Example for a screenshot of how this looks. Thebuild_graphfunction requires you to havegraphviz installed.TargetsInpipecutter.targetsyou find a few targets which build on luigi'sLocalTargetbut additionally have aloadand adumpmethod. A convenient way to name the targets is hereby to use thetask_idin the name, which is unique with respect to the task name and its passed in parameters.importluigiimportpipecutterfrompipecutter.targetsimportJoblibTargetfromsklearn.ensembleimportRandomForestClassifierclassTrainModel(luigi.Task):n_estimators=luigi.IntParameter()defoutput(self):returnJoblibTarget(self.task_id+".joblib")defrun(self):model=RandomForestClassifier(n_estimators=self.n_estimators)self.output().dump(model)pipecutter.run(TrainModel(n_estimators=100))# -> Produces a file called TrainModel_100_0b0ec0cdea.joblibIf you usetask_idin the filename the above task can be written more concise with thepipecutter.targets.outputsdecorator which adds theoutputmethod. By default it puts the files in a folder calleddata. This can be adjusted by the optionalfolderargument.frompipeline.targetsimportoutputs@outputs(JoblibTarget)classTrainModel(luigi.Task):n_estimators=luigi.IntParameter()defrun(self):model=RandomForestClassifier(n_estimators=self.n_estimators)self.output().dump(model)Full exampleimportluigiimportpandasaspdimportnumpyasnpimportpipecutterfromluigi.utilimportrequiresfrompipecutter.targetsimportoutputs,JoblibTarget,ParquetTargetfromsklearn.ensembleimportRandomForestClassifier@outputs(ParquetTarget)classPrepareData(luigi.Task):drop_missings=luigi.BoolParameter()defrun(self):train_df=pd.DataFrame.from_dict({"A":[0,1,np.nan],"B":[5,1,2],"label":[0,1,1]})ifself.drop_missings:train_df=train_df.dropna()self.output().dump(train_df)@requires(PrepareData)@outputs(JoblibTarget)classTrainModel(luigi.Task):n_estimators=luigi.IntParameter()defrun(self):train_df=self.input().load()X,y=train_df.drop("label",axis=1),train_df["label"]model=RandomForestClassifier(n_estimators=self.n_estimators)model.fit(X,y)self.output().dump(model)train_model=TrainModel(n_estimators=100,drop_missings=True)pipecutter.build_graph(train_model)The last command can be used to visualize the dependency tree, which is especially useful if your pipelines are more complex. It returns agraphviz.Digraphobject which will render in a Jupyter Notebook asFinally, run the tasks with:pipecutter.run(train_model)