Package: kedro-to-dataiku
Kedro to DataikuConvert Kedro project to Dataiku project in minutesThis is a tool to enable one to deploy a Kedro (>=0.16.5) project on Dataiku DSS instance without modifying the original Kedro project at all.AutomaticFastFlexibleFeaturesCreate Dataiku datasets automatically based on Kedro dataset catalogConvert Kedro nodes into Dataiku recipesConvert Kedro pipelines into Dataiku flowCreate flow zones in Dataiku project based on Kedro pipeline segmentationLoad all raw input data for the Kedro project into corresponding datasets in Dataiku proejectSupport PySpark through PySpark recipes in DataikuSupport source code clone from git repositoryEnable code editing via Dataiku project libraryAdaption of Kedro projectAs Dataiku flow is basically pandas dataset based, and every single Kedro node will be converted to Dataiku recipe, it is recommended to make inputs and ouputs of Kedro nodes in dataframe format. Pandas DataFrame, PySpark DataFrame, dictionary of Pandas DataFrame will be saeved into Dataiku datasets which can be previewed, while other types of inputs/outputs (array, string, dictionary, etc.) will be saved in managed folders as pickle object.Nodes in Kedro must have distinct function names.As Dataiku recipe must have at least one output, it is recommened to make sure that each Kedro node has at least one output too. However, just in case some nodes in Kedro do not explicitly have any output, this tool will automatically create dummy dataset outputs which are actually not meaningful.If there are local data files, it is required to define the file paths in data catalog configuration either in static format or in templated format (https://kedro.readthedocs.io/en/stable/kedro.config.TemplatedConfigLoader.html) with a macro named as ${data_prefix}. Example:### data locationdata--01_raw--sales.csv### In conf/base/catalog.yml:sales:<<:*csv_tabfilepath:${data_prefix}/01_raw/sales.csv### in conf/local/globals.yml:data_prefix:data/### or simplysales:<<:*csv_tabfilepath:data/01_raw/sales.csvIn order to create Dataiku zones in flow automatically, there must be pipepline segmentation defined in context.pipelines. The keys of context.pipelines can be used to define zones. For example, if there are pipelines defined as the following, then we can use  ["int","primary","master","modeling"] to define the zones.return{"int":int,"primary":primary,"master":master,"modeling":modeling,"__default__":(int+primary+master+modeling),}InstallationAs the package depends on dataiku which is internal module in DSS instance, it is recommneded to install and use this package inside Dataiku DSS.Install it in Dataiku DSS code enviroment like any other pip packages, or install in Jupyter notebook by%pipinstallkedro_to_dataikuThe required packages "dataiku" and "kedro" (>=0.16.5) will be the ones already exist in the Dataiku DSS environment.Instead of installing the package, one can also upload the kernel filehttps://github.com/ppvastar/kedro_to_dataiku/blob/main/kedro_to_dataiku/kedro_to_dataiku.pyto Dataiku project librarylib/pythonso that one canfromkedro_to_dataikuimport*in project code.UsageCreate a managed folder in Dataiku project. Let us suppose it to be "workspace".Compress (into zip) and upload the whole Kedro project root folder (containing subfolders like data, conf, src, etc) into the managed folder, and uncompress it there.Open Jupyter notebook in Dataiku, follow the following steps:Initial set upimportdataiku
fromkedro_to_dataikuimport*### the absolut path to the Kedro project root folder in Dataiku DSS filesystem.kedro_project_path=dataiku.Folder("workspace").get_path()+"[relative path of the kedro project root folder]"### package_name: name of the folder in "[kedro project root folder]/src/" which contains "nodes" and "pipelines" subfolderspackage_name="[Kedro project package name]"### set dataset connection (location). Or any other established connections (like S3) in Dataiku DSS.connection="filesystem_managed"[oranyotherestablishedconnections(likeS3)inDataikuDSS]### data foramt in Dataiku datasetformat_type="csv"### define recipe type. Or use "pyspark" if want to create pyspark recipes.recipe_type="python"### use source code residing in kedro_project_path+"/src". Otherwise, if True, will use source code imported as Dataiku python library -- this option will enable us to edit the soruce code residing in library.src_in_lib=False### a list of zones to be created. They are from the keys of context.pipelines in the Kedro project. Example: ["int","primary","master","master_ds","modeling"]. Or just keep it as None so that no zones will be created automatically.zone_list=None### if want to load the raw input data to Dataiku datasets.load_data=False### if some inputs/outputs of Kedro projects are not Pandas dataframe/Spark dataframe/dictionary of Pandas dataframe format, they will be saved in managed folders instead of Dataiku datasets. This is critical to clarity.folder_list=NoneFast creation and clean## fast creation and clean### one command to create the projectscreate_all(kedro_project_path,package_name,connection,recipe_type,folder_list,zone_list,load_data,format_type,src_in_lib)### one command to clean the projects. Make sure not to delete the managed folder hosting the Kedro project.delete_all(excluded=["workspace"])``Create the project step by step### update data catalog configurationupdate_catalog_conf(kedro_project_path)### create datasetsinput_list,dataset_list=create_datasets(kedro_project_path,package_name,connection,folder_list,format_type,src_in_lib)### create recipescreate_recipes(kedro_project_path,package_name,folder_list,recipe_type,src_in_lib)### create zonescreate_zones(zone_list,folder_list,kedro_project_path,package_name,src_in_lib)### load raw input datasetsload_input_datasets(input_list,kedro_project_path,package_name,src_in_lib)Try some other tools:### list all datastesact_on_project(target="dataset",cmd="list")### clear data in all datastesact_on_project(target="dataset",cmd="clear",excluded=None)### delete all datastesact_on_project(target="dataset",cmd="delete",excluded=None)### return all recipesact_on_project(target="recipe",cmd="list")### delete all recipesact_on_project(target="recipe",cmd="delete",excluded=None)### return all zonesact_on_project(target="zone",cmd="list")### delete all zones except the "Default". Caution: do not delete this Default zone otherwise the project flow will corrupt.act_on_project(target="zone",cmd="delete")### return all foldersact_on_project(target="folder",cmd="list")### delete all foldersact_on_project(target="folder",cmd="delete",excluded=['workspace'])In Dataiku, the src code in managed folder is not editable. If one want to do simple and fast edit on code within dataiku after deployment, one can import the source code to project library (https://doc.dataiku.com/dss/latest/python/reusing-code.html) which is editable. To do this, just load (one can use git) the folder in "[kedro project root folder]/src/" which contains "nodes" and "pipelines" subfolders into the lib/python path (keep the module name as the kedro package name), and then setsrc_in_lib=Truein previouly mentioned steps.By doing so, the soruce code (nodes, pipelines, etc) in this library "lib/python/[package_name]" will be used instead of the orgin one under "[kedro project root folder]/src/[package_name]"One can also clone Kedro project from git repository to the managed folder we created previously.git_url="[git repository URL]"kedro_project_path_in_git="[relative path of Kedro project root folder on git repository]"### Keep existing data folder under the kedro_project_path in Dataiku managed folderclone_from_git(kedro_project_path,git_url,kedro_project_path_in_git)When code has been edited under path lib/python/[package_name], one may want to copy it back to the managed folder (for further download or other operations). To do this, one can use copy_lib function:### overwrite=False, module lib/python/[package_name] will be copied to a new folder: [kedro_project_path]/src/[package_name]_libcopy_lib(kedro_project_path,package_name,overwrite=False)### overwrite=True, [kedro_project_path]/src/[package_name] in managed folder will just be overwritten with the lib/python/[package_name]copy_lib(kedro_project_path,package_name,overwrite=True)ExampleTaking the Kedro 0.17.0 IRIS starter project as an example (https://github.com/ppvastar/kedro_example_iris)One can create a "workspace" managed folder on dataiku, and then upload the project root folder "iris" into the "workspace".Then run code like the following in Dataiku project jupyter notebook:fromkedro_to_dataikuimport*
importdataikukedro_project_path=dataiku.Folder("workspace").get_path()+"/iris"package_name="iris"## change the connection according to actual situationconnection="S3_DSS"recipe_type="python"src_in_lib=Falseload_data=Trueformat_type="csv"folder_list=["example_model","example_predictions"]zone_list=["ds","de"]create_all(kedro_project_path,package_name,connection,recipe_type,folder_list,zone_list,load_data,format_type,src_in_lib)As a magic, the Dataiku flow is created and raw input data is loaded in seconds:The flow is already ready for execution. Note that this simple Iris example is not a typical example, as at the final end there is actually no output (except some log information to report the model accuracy). As a resullt, the end of the Dataiku low is a dummy dataset called as "report_accurary_dummy_output" which will not be utilized.To clear everything just now generated,delete_all(excluded=["workspace"])