Building an Automated Machine Learning Pipeline: Part Four | by Ceren Iyim | Towards Data Science

Part 1: Understand, clean, explore, process data
Part 2: Set metric and baseline, select and tune model
Part 3: Train, evaluate and interpret model
Part 4: Automate your pipeline using Docker and Luigi (you are reading now)
Disclaimer: This article series is not a tutorial about Docker and Luigi. It is the last article of an article series “Building an Automated Machine Learning Pipeline” that focuses on building end-to-end ML pipeline and showing how to automate it using certain elements of both tools. This article will make more sense to you if you read the previous ones from the links above.
In this article series, we set our course to build a 9-step machine learning (ML) pipeline and automate it using Docker and Luigi.
Understand & Clean & Format DataExploratory Data AnalysisFeature Engineering & Pre-processingSet Evaluation Metric & Establish BaselineSelect an ML Model based on the Evaluation MetricPerform Hyperparameter Tuning on the Selected ModelTrain and Evaluate the ModelInterpret Model PredictionsDraw Conclusions & Document Work
Understand & Clean & Format Data
Exploratory Data Analysis
Feature Engineering & Pre-processing
Set Evaluation Metric & Establish Baseline
Select an ML Model based on the Evaluation Metric
Perform Hyperparameter Tuning on the Selected Model
Train and Evaluate the Model
Interpret Model Predictions
Draw Conclusions & Document Work
As a result of this pipeline, we built our ML solution and called it, the wine rating predictor because we are trying to infer the quality of wine represented with the points using a sample dataset. We defined the requirements for our wine rating predictor in the first article as:
- understandable because our audience will likely have limited knowledge of statistics and ML.
- performant because the complete production dataset could have millions of rows.
- automated in a way that it can run on any production system without requiring specialized configurations and setups.
We have so far satisfied the understandability and addressed performance — to some extent:
We have selected an understandable model, random forest regressor, as the underlying ML algorithm of the wine rating predictor. Moreover, we have explained how random forest regressor works as well as its hyperparameters in the second and third article.
We have built a performant model by performing hyperparameter tuning on the random forest regressor. We still have room for it though.
Today, we will wear a software engineer’s hat and address the last requirement — automation. We will take the steps below from the ML pipeline, run them on Docker Containers and connect them with Luigi Tasks. (Don’t worry we will elaborate on them throughout the article 🙂)
If you have noticed we have two additional steps because a typical real-world ML pipeline starts with getting the data from a source. Also, we included the training-test dataset split and the rest is the known steps of the ML pipeline.
To run the flow above, we will first look at Docker and Luigi and understand their use in the context of this article series. Therefore, we will start by explaining these tools and their necessary elements.
Then, we will connect the dots using these tools. Eventually, we will put the pieces together and run the flow which will accomplish our mission of automation!
What Is Docker and How It Is Used for the Wine Rating Predictor?What Is Luigi and Why It Is Selected as an Orchestration Tool?Put the Pieces of Docker and Luigi TogetherRun and Automate the Pipeline!
What Is Docker and How It Is Used for the Wine Rating Predictor?
What Is Luigi and Why It Is Selected as an Orchestration Tool?
Put the Pieces of Docker and Luigi Together
Run and Automate the Pipeline!
You can find the GitHub Repository here:
github.com
One more disclaimer before we start:
Disclaimer: Structure of the repository and download_data.py , util.py , docker-clean.sh and docker-compose.yml files are provided to me as the basis of the code challenge. Rest of the code is written by me.
The official definition of Docker from Wikipedia is as follows:
Docker is a set of platform as a service products that uses operating system-level virtualization to deliver software in packages called containers.
I would define it as creating virtual environments for your application/software/data science/ML projects so that it can run seamlessly on any system that has Docker.
Via containers. They are isolated virtual environments to run your applications, for our case, ML solutions.
From images. They are templates that are used to create containers. They contain information about your underlying operating system, environment and libraries — with versions.
By reading the instructions from a Dockerfile. It is a set of instructions and commands that a user could call on the command line to create an image. You can create a local image in your system with the docker build Dockerfile -t <image_name> command.
Containers are the running instances of images. When you run an image you create a container that is an isolated environment.
We used Docker to build virtual environments and manage the dependencies of the libraries so that we can run the flow seamlessly. The automation flow below is designed so that each box runs on a separate Docker container.
I am sure you have noticed some directories and files in the repository that we haven’t mentioned yet. You can think of download_data, make_dataset, clean_data, extract_features, transform_data, impute_data, train_model and evaluate_model directories as the boundaries of the boxes in the automation flow that contains a Dockerfile and the source code as a Python file.
Before looking at how to create a Dockerfile, let’s understand the use of base_docker here.
Each step of the automation flow runs on a separate container. However, they are modules written in Python and the only difference between them are the libraries used. So, base_docker is used to specify the common environment variables and install the required Python version and the libraries. It also contributed to the performance of the wine rating predictor by installing required packages once rather than installing them every time on a separate container.
When creating a Dockerfile start by the steps as if you are creating a local environment on your machine. Considering your aim is to deploy an ML solution with Python, you can build your image on top of an existing Python Image in the Docker Hub. An important point here is to select an existing Python Image that is suitable for your case since there are several versions and sizes are available.
FROM python:3.7-slimENV LC_ALL=C.UTF-8ENV LANG=C.UTF-8ENV PYTHONPATH=/opt/orchestrator:$PYTHONPATHCOPY requirements.txt /opt/base_docker/RUN pip install -r /opt/base_docker/requirements.txtWORKDIR /opt/base_docker/
slim variant is chosen and Python version 3.7 is specified, same Python version as in my local system.
ENV command is used to update the PATH environment variable for the underlying operating system (Python 3.7-slim for our case) that your container installs.
Libraries with the versions mentioned in the requirements.txt are copied and installed with the pip_install. In addition to the aforementioned libraries of Python, click is also installed — more on the use of that later.
As the last instruction, the working directory of a Docker Container is defined with the WORKDIR command.
Now, our base_docker is ready, exploring any Dockerfile from the automation flow will be sufficient because all other Dockerfiles have been written with the same logic. Here is an example from clean_data:
FROM code-challenge/base-dockerCOPY . /opt/clean_data/WORKDIR /opt/clean_data/
Now our base image become the base_docker. Every file in the clean_data directory is copied and a working directory of a Docker container is created.
It is time to have a look at Spotify’s beloved Luigi!
The official definition from the docs is as follows:
Luigi is a Python package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.
It is basically an orchestration tool to stitch many tasks together in a directed acyclic graph (DAG) structure:
When you build an ML solution and as the scale grows, this can easily become complex and messy. Luigi is a robust workflow management tool that can prevent chaos and complexity.
There are two fundamental building blocks of Luigi: Task and Target
The steps of a workflow are tasks, usually a single unit of work, where computations are done. A Luigi workflow is built on top of tasks.
Each task is connected with a target. A target can be a file, a checkpoint in the workflow or any kind of output generated by the task.
We are dumping outputs generated by each task in the data_root:
In the flow, each task is dependent on the output of the previous task except the initial task download_data. Each file is passed as a Parameter so that it can be consumed by the latter step in the workflow.
When designing a workflow, Luigi recommends the atomic structure: Each task should have a single file as output like in the download_data and train_model tasks.
For the rest of the tasks, we imitate atomicity with the SUCCESS flag. When we perform computations on both training and test datasets, we create a SUCCESS flag as an output of the task, after dumping the output files in the directory. This way, the next task only checks whether a SUCCESS flag exists.
Task dependencies are defined with the requires() method. Here is a task outline from Luigi documentation:
For our case, task dependencies are obvious since we defined a linear automation flow.
Unfortunately, Luigi does not come with a triggering mechanism. If you want to run a Luigi workflow you use the command line specifying the module name and the last task in the project directory:
luigi --module <modul_name> <task_name> 
When a workflow is triggered, Luigi checks if the output of the last step exists. If not, then it checks backwards if the outputs of the predecessor steps exist. For our case, this would be train_model, impute_data, transform_data, extract_features, clean_data, make_dataset and download_data .
If an output of a task from any step of the flow exists Luigi resumes the flow where it has left of. This is a very useful and important trait to prevent your ML pipeline from crashing when it contains partial data.
Now we know the essential elements of Luigi and Docker we can move forward with completing the picture!
We will put the pieces of Docker and Luigi in the docker-compose.yml and orchestrator directory.
The utility functions provided by the awesome engineers of Data Revenue has enabled me to connect the pieces of Docker and Luigi. Functions and classes available in the util.py played an important role that connected Docker and Luigi for the wine rating predictor. For example, DockerTask — the object that we pass as a parameter between the tasks — is not available as a parameter in the Luigi normally.
First, let’s have a look at task.py in the orchestrator directory where I built the automation flow using utilities. Then, let’s understand the purpose of docker-compose.yml.
Luigi Parameter: Each task starts with a LuigiParameter object where we specify the input/output files and directories of the task. For example, the last task EvaluateModel where we evaluate the ML model with the test set takes the test_features,test_target and the trained model file (model.sav) as an input LuigiParameter. Since this task creates multiple files as output (PredictionsVSActuals.png and FeatureImportances.png) we also define a SUCCESS flag to mimic atomicity.
Image: We define which Docker Image to use to create the container.
Dependent Task: If there are any dependent tasks, define them here with the requires() method. The dependent task of the EvaluateModel is TrainModel.
Command: The command to be executed by the container. Here command calls the source code available in the evaluate_model.py with the relevant parameters.
Output: The output() method returns one or more Target objects (either the output file or a SUCCESS flag)
Recall that source codes for each step of the flow is available in the respective directory as Python file. Let’s have a look at the source code of train_model :
After defining the helper functions convert_features_to_array and convert_target_to_array, we defined necessary click commands.
Click, as defined in the official documentation:
It is a Python package for creating beautiful command line interfaces in a composable way with as little code as necessary.
We used Click to be able to pass the arguments to the python scripts from the command line. Finally, we train the model with the fine-tuned parameters that we have decided in the Perform Hyperparameter Tuning on the Selected Model step on the training dataset. Finally, we save the trained model in model.sav file as an output.
The official definition for Docker Compose from the Docker Docs is as follows:
Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.
This definition suits to our case since we run the automation flow on 9 containers including base_docker. You can read more about the use and commands of docker-compose here.
The important part for us here the command to run Luigi tasks is written here, so when we run containers through docker-compose, it will trigger the automation flow with the luigi — module task EvaluateModel-scheduler-host luigiid command.
This is the moment of truth! We are going to run the automation flow and automate the 9-Step ML Pipeline. It is the last of the last step, stick with it for one more minute 😉
For me this is:
cd GitHub/Wine-Rating-Predictor-ML-Model
Instead of building multiple Docker Containers one by one, we are going to use a shell script to build Docker Containers: build-task-images.sh
./build-task-images.sh 0.1
The below message shows that Containers are built successfully. So we are good to go for triggering automation flow.
Successfully tagged code-challenge/evaluate-model:0.1
We will trigger the workflow with
docker-compose up orchestrator
Upon writing this command, Luigi checks whether each task is complete and outputs the following information:
Checking if EvaluateModel(    no_remove_finished=False,    in_test_features_csv=/usr/share/data/interim/test_features.csv,    in_test_target_csv=/usr/share/data/interim/test_target.csv,    in_trained_model=/usr/share/data/output/model.sav,    out_dir=/usr/share/data/output/,     flag=.SUCCESS_EvaluateModel) is completeChecking if TrainModel(   no_remove_finished=False,   in_train_features_csv=/usr/share/data/interim/train_features.csv,   in_train_target_csv=/usr/share/data/interim/train_target.csv,   out_dir=/usr/share/data/output/) is completeChecking if ImputeData(    no_remove_finished=False,    in_train_csv=/usr/share/data/interim/train_transformed.csv,    in_test_csv=/usr/share/data/interim/test_transformed.csv,    out_dir=/usr/share/data/interim/,     flag=.SUCCESS_ImputeData) is completeChecking if TransformData(   no_remove_finished=False,  in_train_csv=/usr/share/data/interim/train_features_extracted.csv,  in_test_csv=/usr/share/data/interim/test_features_extracted.csv,  out_dir=/usr/share/data/interim/,   flag=.SUCCESS_TransformData) is completeChecking if ExtractFeatures(    no_remove_finished=False,    in_train_csv=/usr/share/data/interim/train_cleaned.csv,    in_test_csv=/usr/share/data/interim/test_cleaned.csv,    out_dir=/usr/share/data/interim/,     flag=.SUCCESS_ExtractFeatures) is completeChecking if CleanData(    no_remove_finished=False,    in_train_csv=/usr/share/data/interim/train.csv,    in_test_csv=/usr/share/data/interim/test.csv,    out_dir=/usr/share/data/interim/,     flag=.SUCCESS_CleanData) is completeChecking if MakeDatasets(    no_remove_finished=False,    in_csv=/usr/share/data/raw/wine_dataset.csv,    out_dir=/usr/share/data/interim/,     flag=.SUCCESS_MakeDatasets) is completeChecking if DownloadData(    no_remove_finished=False,     fname=wine_dataset,     out_dir=/usr/share/data/raw/,    url=https://github.com/datarevenue-berlin/code-challenge-2019/releases/download/0.1.0/dataset_sampled.csv) is complete
Then runs the task in one by one. Another important information for us is:
INFO:evaluate-model:Mean square error of the model is: 4.95
After a successful run you see a 🙂:
Bonus Points: Let’s see how Luigi resumes the flow where it has left ofLet’s say you prepared training datasets and run the automation flow, then you will build the next steps afterwards starting with the training the model. Luigi won’t repeat successfully run previous steps and only runs TrainModel and EvaluateModel tasks in that case.
I would say Wow!
It was quite a journey spanning over 4 articles and a month ⛵️
We started from the very beginning and analyzed every step of the 9-step ML pipeline through the first, second and third article.
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
In this last article, we introduced the relevant elements of Docker and Luigi and explained their importance and use for the wine rating predictor. We completed the last missing piece of our comprehensive pipeline and did a dry run of an ML solution — just as if it runs on production systems.
I think it is time to thank my significant other Serkan Durusoy. He has been the most precious mentor, editor and study buddy I had in my data science expedition!
He helped me to overcome the ups and downs of changing a career. Also, I want to acknowledge his support to complete this article series as well as the grit he provided for me. Thank you, my dear ❤️
Thanks for reading 🙂 Please feel free to use this pipeline, the code and the repository for your own projects.
For comments or constructive feedback, you can reach out to me on responses, Twitter or Linkedin!
Before we go, I also want to share some of the useful resources that I used to prepare for this project. See you in the next article 👋
Scale Your Machine Learning Pipeline by Data Revenue
Docker for Beginners: Full Course by KodeKloud
Build a Docker Container with Your Machine Learning Model by Tina Bu
Luigi Documentation
Docker Documentation
Building Data Science Pipelines With Luigi and Jupyter Notebooks by Mattia Ciollaro