Kedro — A Python Framework for Reproducible Data Science Project
By: Khuyen Tran
Tags: Data Science, Pipeline, Python, Machine Learning, Editors Pick

You might try to save the data then check it in your Jupyter Notebook to make sure the output is as you expected. This approach works, but it is cumbersome.

Another common issue is that it’s hard to understand the relationships between functions when looking at a Python script that contains both the code to create and execute functions.

Your code looks even more complex and hard to follow as the project grows.

Wouldn’t it be nicer if you can visualize how the inputs and outputs of different functions are connected like below?

Image by Author

That is when Kedro comes in handy.

What is Kedro?

Kedro is an open-source Python framework for creating reproducible, maintainable, and modular data science code. It borrows concepts from software engineering best-practice and applies them to machine learning code.

Kedro allows you to:

Create a data science from a cookie-cutter template Create a data science pipeline Slice a pipeline Modularize a pipeline Configure your data and parameters through a YAML file Effortlessly analyze nodes’ outputs in a Jupyter Notebook Visualize your pipeline Create documentation for your project

In this article, I will go through each of these features and explain how they can be helpful for your data science projects.

To install Kedro, type:

pip install kedro

Set Up a Data Science Project

Create a Data Science from a Cookie-Cutter Template

Have you ever contemplated how to structure your data science project so that it is logical and reasonably standardized? Wouldn’t it be nice if you can create a well-defined and standard project structure in one line of code?

That could be easily done with Kedro. After installing Kedro, you can create a new empty project using:

$ kedro new

After going through a series of questions, a new project will be created with the structure like below:

Image by Author

If we look at the project structure at a higher level, we can see that there are 5 main directories:

Image by Author

conf : Store configuration files

: Store configuration files data : Store data

: Store data docs : Store documentation of the project

: Store documentation of the project logs : Store log files

: Store log files notebooks : Store Jupyter Notebooks

: Store Jupyter Notebooks src : Store the main code

Install Dependencies

Kedro requires some basic dependencies before using. These dependencies are specified under src/requirements.txt . To install these dependencies, type:

$ kedro install

And all necessary dependencies to run the pipelines will be installed in your environment.

Now that we learn how to set up a data science project, let’s understand how to create a pipeline with Kedro.

Create a pipeline

To create a new pipeline, type:

$ kedro pipeline create <NAME>

Since there are often 2 steps in a data science project: data processing and model training, we will create a pipeline called data_engineer and a pipeline called data_science :

$ kedro pipeline create data_engineering

$ kedro pipeline create data_science

These two pipeline directories will be created under src/project_name :

Image by Author

Each pipeline consists of 4 files:

__init__.py

README.md : specifies information about the pipeline

: specifies information about the pipeline node.py : contains nodes

: contains nodes pipeline.py : contains pipelines

Node

A pipeline consists of multiple nodes. Each node is a Python function.

For each node, there are input(s) and output(s):

Visualization of the node get_classes :

Image by Author

Inputs and outputs of each node can be specified using None , string, a list of strings, or a dictionary. Note that these strings are abstract names, not the real values.

Why are names useful? If we know the name of each function’s inputs and outputs, we can easily grab a specific input or output by calling its name. Thus, there is less ambiguity in your code.