Package: kedro-dataframe-dropin
kedro-dataframe-dropinHow do I get started?$pipinstallkedro-dataframe-dropin--upgradeThen what?Replace yourpandas.*DataSetin yourcatalog.ymlwithkedro_dataframe_dropin.[rapids|modin].*DataSetand reap the benefits, as long as your node and pipeline code is compatible with thecudf/modinAPI (that tries to replicatepandasas much as possible) and your data format is supported by the respective libraries (for example,cudfdoesn't support theread_excelmethod)What is kedro-dataframe-dropin?kedro-dataframe-dropin is a Kedro plugin that provides modified versions of thepandas.*dataset definitions (e.gpandas.CSVDataSet) from Kedro, where each dataset has been replaced with one ofpandasdrop-in replacements.For examplekedro_dataframe_dropin.modin.CSVDataSetreplicatespandas.CSVDataSetbut with themodin.pandaspackage replacingpandas. Likewise,kedro_dataframe_dropin.rapids.CSVDataSetprovides acuDF-backed version of theCSVDataSet.Why does this exist?There might be several reasons why you'd want to consider a drop-in replacement for Pandas. The use-cases are outlined in various places, such as: themodin documentationorthe RAPIDS website.However, the only dataframe-backed datasets that Kedro has out of the box are thepandasandpysparkones. If you wanted to use, say, amodindataframe backed byDaskorRay, you'd need to write acustom datasetfor each file format (.csv,.xls, etc...).This lets you swap out yourcatalog.ymlfrom:# conf/base/catalog.yml [before]rockets:type:pandas.CSVDataSetfilepath:data/01_raw/rockets.csvreviews:type:pandas.ExcelDataSetfilepath:data/01_raw/reviews.xslsxto:# conf/base/catalog.yml [after]rockets:type:kedro_dataframe_dropin.rapids.CSVDataSetfilepath:data/01_raw/rockets.csvreviews:type:kedro_dataframe_dropin.modin.ExcelDataSetfilepath:data/01_raw/reviews.xlsxand as long as the code within your nodes fits withinmodinorcudf's implementation of a subset of thepandasAPI, you'll be done!What dropins are currently supported?dropinsupportedmodin[ray]✅modin[dask]✅cudf✅dask🟠dask-cudf🟠✅: compatible
🟠: No kedro versioning and some datasets (likeSQLTableDataSet) don't work despite being available on bothkedroand the drop-in.What happens when Kedro adds or changes apandasdataset?The beauty of it is that this will stay in complete sync with Kedro'spandas.*library without any code changes or releases required. It's implemented through hot-swapping thepandasmodule with one of the replacements you specified.ExamplesAs an example of why you might want to use this, here are the results of some very rough and preliminary benchmarking. These were conducted on a Google Colaboratory notebook (thanks Google!) with a Tesla T4 GPU and a 2-core CPU. The data used was a 5 million row CSV, weighing in at around a 100mb downloaded fromhere.# base/conf/catalog.yml
cudf:
  type: kedro_dataframe_dropin.rapids.CSVDataSet
  filepath: data/01_raw/data.csv

pandas:
  type: pandas.CSVDataSet
  filepath: data/01_raw/data.csvUsing the two datasets within thekedro ipythonconsole shows a world of difference, with reading the file in being 10x faster, doing a groupby being 6x faster and taking the mean being 5x faster.This helps shorten:The feedback loop when prototyping and exploring your data within akedro ipythonor akedro jupytersessionThe feedback loop when running your pipelines in development and debugging/experimenting with various different methodologiesYour production runtimeIn [1]: %timeit gdf = catalog.load("cudf")
702 ms ± 7.32 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [2]: %timeit df = catalog.load("pandas")
8.22 s ± 101 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [3]: %timeit gdf.groupby("Region")
4.75 µs ± 56.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [4]: %timeit df.groupby("Region")
26.7 µs ± 397 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [5]: %timeit df["Total Revenue"].mean()
11.8 ms ± 87.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [6]: %timeit gdf["Total Revenue"].mean()
2.71 ms ± 31.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)Any additional benchmarks you do and want to share back would be much appreciated. Feel free to open an issue!Some special notes on RAPIDSThe rest of thecu*ecosystemYour data processing step gets faster (assuming you have the right conditions) by plugging in thecudfmodule from RAPIDs in place ofpandas, but it doesn't end there.You can continue to make use of your GPU speedup in the rest of your pipeline lifecycle (predictions, ML, graph, etc...) by using the rest of thecudaecosystem of tools (cuMLand the ilk) in place of tools likesklearn.Why are some data formats missing?With the way this plugin was designed, it only hot swaps incudfin place ofpandaswhere the Kedro pandas dataset exists.So as it stands today, with the Kedro codebase not having anORCDataSetfor example, this plugin won't have it either. You'll need to build your own custom own.Or better yet, head over to theKedrocodebase and contribute thepandasversion of it to their codebase. This plugin will then automatically pick it up and provide acudf-equivalent.Some special notes ondask-cuDFanddaskNote thatdaskanddask-cuDFwill delay compute and operations across nodes are actually building up a computation graph. They will be parallelised across your CPU/GPU when you invoke a.compute()operation (likelenor save it to disk by having its output be a non-memory dataset in the catalog).Note that Kedro versioning won't be possible with these datasets, since Kedro completely owns the I/O and simply passes the file handle down todask/dask-cuDFwhich doesn't accept it - since file handles can't be shared across (CPU or GPU) workers. Instead what we do is extract the filepath and pass it todaskwho also usefsspecand so you still have full remote-layer interopability with the benefit of parallelised compute.Consider giving Matthew Rocklin'sblog post ondask-cuDFand the philsophy of it simply being a different "engine" fordask.DataFramea read.CaveatsKeep in mind that in order to remain consistent with the adage of not copying memory, when passing these dataframes between nodes, theywill notbe copied - but simply passed through as the same underlying Python object, so if you're doing mutable operations on them across different nodes, keep in that in mind.