Structuring ML Pipeline Projects
By: Theodoros Ntakouris
Tags: Deep Learning, Machine Learning, Tensorflow Extended, Cloud Native, Software Engineering

Directory Structure and Intuition Behind It

$project-name is the root directory of your project

is the root directory of your project $project-name/ml includes machine learning related stuff.

includes machine learning related stuff. $project-name/ml/pipelines includes the actual ML pipeline code

includes the actual ML pipeline code Typically, you may find yourself with multiple ML pipelines to manage, such as $project-name/ml/pipelines/predict-sales and $project-name/ml/pipelines/classify-fraud or similar.

and or similar. Here is a simple tree view:

$project-name/ml/pipelines includes the following:

data → small amount of representative training data to run locally for testing and on CI. That’s true if your system does not have a dedicated component to pull data from somewhere. If this is true, make sure to include a sampling query with a small limited number of items.

→ small amount of representative training data to run locally for testing and on CI. That’s true if your system does not have a dedicated component to pull data from somewhere. If this is true, make sure to include a sampling query with a small limited number of items. util → code that is reused and shared across $pipeline-name s. It is not necessary to include input_fn_utils.py and model_utils.py . Use whatever makes sense here. Here are some examples:

In my own projects, it made sense to abstract some parts on the utility module, like building named input and output layers for the keras models.

Building the serving signature metagraph using Tensorflow Transform output.

Preprocessing features into groups by using keys.

And also other common repetitive tasks, like building input pipelines with the Tensorflow dataset api.

cli.py → entry point and command line interface for the pipelines. Here are some common things to consider when using TFX.

By using abseil you can declare and access flags globally. Each module defines flags that are specific to it. It is a distributed system. This means that the common flags, like --data_dir=... , --hparam_tuning , --pipeline_root , --ml_metadata_url , --use_cache , --train_epochs is some you can define on the actual cli.py file. Other, more specific ones for each pipeline can be defined on submodules.

This file acts as an entry point for the system. It uses contents in pipeline.py to set up the components of the pipeline as well as provide the user-provided module files ( in the tree example these are constants.py , model.py , training.py ) based on some flag like --pipeline_name=$pipeline-name or some other configuration.

Finally, with the assembled pipeline, it calls some _runner.py file, by using a --runner= flag.

pipeline.py → parameterised pipeline component declaration and wiring. This is usually just a function that declares a bunch of TFX components and returns a tfx.orchestration.Pipeline object.

local_beam_dag_runner.py → configuration to run locally with the portable Beam runner. This can typically be almost configuration — free, just by using the BeamDagRunner .

→ configuration to run locally with the portable Beam runner. This can typically be almost configuration — free, just by using the . kfp_runner.py → configuration to run on Kubeflow Pipelines. This typically includes different data path and pipeline output prefixes and auto-binds an ml-metadata instance.