Package: kedro-wings
Kedro WingsAs Seen onYouTube DataEngineerOne:Kedro Wings: It's almost too easy to write pipelines this way.Easy Stateful Pipelines with Chronocoding and Kedro WingsGive your next kedro project Wings! The perfect plugin for brand new pipelines, and new kedro users.
This plugin enables easy and fast creation of datasets so that you can get straight into coding your pipelines.Quick Start Usage Example: Iris ExampleThe following example is a recreation of the iris example pipeline.Kedro Wings enables super fast creation of pipelines by taking care of all the catalog work for you.
Catalog entries are automatically created by parsing the values for your nodes' inputs and outputs.This pipeline automatically creates a dataset that reads from theiris.csvand then it creates 12 more datasets, corresponding to the outputs and inputs of the other datasets.wing_example=Pipeline([node(split_data,inputs=['01_raw/iris.csv','params:example_test_data_ratio'],outputs=dict(train_x="02_intermediate/example_train_x.csv",train_y="02_intermediate/example_train_y.csv",test_x="02_intermediate/example_test_x.csv",test_y="02_intermediate/example_test_y.csv")),node(train_model,["02_intermediate/example_train_x.csv","02_intermediate/example_train_y.csv","parameters"],outputs="06_models/example_model.pkl",),node(predict,inputs=dict(model="06_models/example_model.pkl",test_x="02_intermediate/example_test_x.csv"),outputs="07_model_output/example_predictions.pkl",),node(report_accuracy,inputs=["07_model_output/example_predictions.pkl","02_intermediate/example_test_y.csv"],None),])Quick Start Example: ChronocodingWatch the video on Chronocoding here:Easy Stateful Pipelines with Chronocoding and Kedro WingsSometimes, there arises a need to rewrite data to the same path. This makes it easier to save state between kedro runs.
Using KedroWings, you can automatically generatechronocodeddatasets which temporally separates a read and write to a dataset.By adding an!at the end of a dataset, we signal to kedro that we wish to overwrite the data in that same filepath. Thus, we get around kedro's DAG requirement for datasets.In Depth Breakdown on Chronocoding here:[KED-1667] Chronocoding: Solving the Problem of State Tracking with Temporally Sensitive DAGsdefstate_modifier(state:str)->str:current_value=int(state)new_value=current_value+1returnstr(new_value)defcreate_pipelines(**kwargs):returnPipeline([node(state_modifier,inputs="01_raw/state.txt",outputs="01_raw/state.txt!"),])InstallationKedro Wings is available on pypi, and is installed withkedro hooks.pip install kedro-wingsSetup with Kedro PipelineSimply add aKedroWingsinstance to theProjectContexthookstuple.fromkedro_wingsimportKedroWingsclassProjectContext(KedroContext):hooks=(KedroWings(),)Setup with Jupyter NotebookSimply pass the kedro context intoKedroWings, and it will automatically add all catalog entries from all available pipelines.# Load the context if not using a kedro jupyter notebookfromkedro.framework.contextimportload_contextcontext=load_context('./')# Pass the context into KedroWingsfromkedro_wingsimportKedroWingsKedroWings(context=context)# context catalog now has all wings datasets available.context.catalog.list()UsageCatalog CreationCatalog entries are created using dataset input and output strings. The API is simple:inputs="[PATH]/[NAME].[EXT]"ThePATHportion determines the directory where a file will be saved.
TheNAMEportion determines the final output name of the file to be saved.
TheEXTportion determines the dataset used to save and load that particular data.Ex: Creating an iris.csv readernode(split_data,inputs='01_raw/iris.csv',outputs='split_data_output')This will create apandas.CSVDataSetpointing at the01_raw/iris.csvfile.Ex: Overwrite a Kedro Wing dataset usingcatalog.yml# pipeline.pynode(split_data,inputs='01_raw/iris.csv',outputs='split_data_output')# catalog.yml01_raw/iris.csv':type:pandas.CSVDataSetfilepath:data/01_raw/iris.csvIf a catalog entry already exists inside ofcatalog.yml, with a name that matches the wing catalog name,
KedroWings will NOT create that catalog, and will instead defer to thecatalog.ymlentry.Default DatasetsThe following are the datasets available by default.default_dataset_configs={".csv":{"type":"pandas.CSVDataSet"},".yml":{"type":"yaml.YAMLDataSet"},".yaml":{"type":"yaml.YAMLDataSet"},".xls":{"type":"pandas.ExcelDataSet"},".txt":{"type":"text.TextDataSet"},".png":{"type":"pillow.ImageDataSet"},".jpg":{"type":"pillow.ImageDataSet"},".jpeg":{"type":"pillow.ImageDataSet"},".img":{"type":"pillow.ImageDataSet"},".pkl":{"type":"pickle.PickleDataSet"},".parquet":{"type":"pandas.ParquetDataSet"},".json":{"type":"json.JSONDataSet"},# Only available in kedro 0.16.3}ConfigurationKedro Wings supports configuration on instantiation of the hook.KedroWings(dataset_configs, paths, root, namespaces, enabled, context)dataset_configs:param dataset_configs: A mapping of file name extensions to the type of dataset to be created.This allows the default dataset configurations to be overridden.
This also allows the default extension to dataset mapping to be overridden or extended for other datasets.Longer extensions are prioritized over shorter extensions,
meaning multiple encoding methods can be applied to a single filetype.Ex: Make default csv files use pipes as separatorsKedroWings(dataset_configs={'.csv':{'type':'pandas.CSVDataSet','sep':'|'},})Ex: Use dataset types directlyfromkedro.extras.datasetimportpandasKedroWings(dataset_configs={'.csv':pandas.CSVDataSet,})Ex: Save CSVs with pipes or commasfromkedro.extras.datasetimportpandasKedroWings(dataset_configs={'.comma.csv':pandas.CSVDataSet,'.pipe.csv':{'type':'pandas.CSVDataSet','sep':'|'},})pathsThis allows specified paths to be remapped:param paths: A mapping of old path names to new path names.Ex: Moving data from 06_models to a new_models folderKedroWings(paths={'06_models':'new_models',})rootThis setting is prepended to any paths parsed. This is useful if the dataset supportsfsspec.:param root: The root directory to save files to. Default: dataEx: Saving data to s3 instead of the local directory.KedroWings(root='s3a://my-bucket/kedro-data')Ex: Allow individual datasets to choose their rootKedroWings(root=None)namespacesNamespaces from modular pipelines are supported.
This parameter should be a list of the namespaces that KedroWings should account for.
If a namespace is encountered, the output filepath will include the namespace in the extension.Ex: NamespaceThe determined file paths would beiris.example1.csvandiris2.example2.csv.KedroWings(namespaces=['example1'])pipeline(Pipeline([node(lambdax:x,inputs='iris.csv',outputs='iris2.csv')]),namespace="example1")enabledThis setting allows easy enabling and disabling of the plugin.:param enabled: Convenience flag to enable or disable this plugin.Ex: Use an environment variable to enable or disable wingsKedroWings(enabled=os.getenv('ENABLE_WINGS'))