Use Docker and Airflow to deploy your Data Science workflow
By: Madan Krishnan
Tags: Blog Post, Big Data, Data Science, Docker, Airflow

Contributors: Madana Krishnan V K, Nguyen Cao, Sanjana Chauhan, Sumukha Balasubramanya

This blog is written and maintained by students in the Professional Master’s Program in the School of Computing Science at Simon Fraser University as part of their course credit. To learn more about this unique program, please visit {sfu.ca/computing/pmp}.

Can a UI designer decide which sorting algorithm to use? Well, you can say it’s a mismatch. But can a Data Scientist deploy his machine learning model at scale? Let us try answering this question through this post.

There are different stakeholders in a Big Data project like Data Scientists, Data Engineers, Analysts, etc. A Data Scientist is one who gets insightful meanings from structured or unstructured data by using various techniques and tools, while a Data Engineer is one who develops and maintains architectures such as databases and large scale processing systems.

Being a Data Scientist is not easy as it involves many fields such as Statistics, Computer Science, and Business Analysis. They are dependent on Data Engineers to make their code work the same way in the production environment. This blog aims at helping a Data Scientist to enhance their understanding of how their code would run in a production environment.

Why think about deployment as a Data Scientist?

A Data Scientist mostly thinks about developing Machine Learning models to provide insights into data or to predict valuable results. The commonly used interface/tool for the Data Science workflow is Jupyter Notebook/Google Colab. These interfaces are well designed for writing and executing code snippets, as they provide an interactive shell to play with. But the downside to this is that it runs on a single machine, and usually does not work well with large datasets and has no parallelism. It is not suitable for running on a production environment with a distributed system running on clusters.

The lack of skill to write a production-ready code among Data Scientists requires a Data Engineer to transform the code into scripts running on production systems parallelly, which takes time and resources. This article aims to provide a mapping of the typical Data Science workflow on a local machine to a sequence of tasks running on servers.

How to go about it?

Let us use Docker and Airflow to achieve this.

Why Docker?

Docker is a tool designed to make it easier to create, deploy, and run applications by using containers. Containers allow a developer to package up an application with all of the parts it needs, such as libraries and other dependencies, and ship it all out as one package. By doing so, the developer can be rest assured that the application will run on any other machine regardless of any customized settings that machine might have that could differ from the machine used for writing and testing the code.

The below matrix consists of possible software components as rows and possible places where you must run that software as the column.

Our job is basically to make sure that every intersection of that matrix somehow works right, passes all the tests on your laptop as well as in the production environment. How to be sure about that?

Suppose your laptop has a different version of Python or different a JDK, but the production environment has a different kind of distribution. So, in a nutshell, what we are trying to do with docker is to solve this kind of problem.

Let us take an example of Shipping Coffee Beans.

The above matrix here depicts the possible goods to be shipped, multiplied by every possible way to ship the goods. It had the same problem.

How did they solve it?

In simple words, if you want to ship Coffee Beans, it should not be your concern to decide the shipping, packaging and routing details and to make sure experts for each of these are available. Docker Container is similar to shipping containers with pre-decided infrastructure (size, number of doors, weights it can take, etc.). Just pack your coffee beans and hand it over to any infrastructure provider.

It can be organized in such a way that any new infrastructure providers or new infrastructure tools can be added. There is no need to repackage the Coffee Beans just because you are not going through the same route or using the same transport. Also conversely, if you are the ones who are providing the infrastructure you can decide on cheaper trucks and faster routes.

Voila!!! The solution to the above problem → “Docker”

Data Scientists spend so much time to build the perfect model and it takes a lot of effort to set up the infrastructure, installing the required libraries choosing the fastest tools. They do not want to spend the same amount of time setting up the same in the production environment or design their solutions as per the available production infrastructure.

To make this idea more powerful, using Airflow, let us see how can a Data Scientist automate their workflow.

Why Airflow?

Airflow is a platform created by the community to programmatically author, schedule and monitor workflows. We use Airflow to author workflows as Directed Acyclic Graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

In order to execute your workflow, you can use Airflow to automate and decide the sequence in which these activities should be scheduled. For example, you want to do web scraping to collect data using Java programming, store the data in Cassandra, build a machine learning model using TensorFlow, etc. All tasks require different infrastructure and should be performed in a sequence. Using Airflow you can schedule these tasks by running different Docker Containers.

Data Science workflow

Data Science workflow in action

The dataset considered here is the MNIST data, which is imported from the Scikit-learn datasets. It consists of 70,000 handwritten digits, where each image is fit into a 28x28 grid. Training a classifier model to predict the handwritten digits is a classical example in the world of Data Science and Machine Learning.

To demonstrate the workflow, we perform the task of training classifiers on the MNIST data. The MNIST data from the Scikit-learn library is collected, preprocessed, split and is saved in the file system. These are the data collection and preprocessing steps. The data is read from the file system, and two Machine Learning models (LogisticRegression and SGDClassifier) are trained and saved. This step is the data analysis step. Then the generated model is used and prediction is performed on the test set. The prediction and visualization procedures of the result are the data visualization step.

DAG Workflow

The above graph depicts the workflow we are demonstrating inside Airflow.

It shows the organization of tasks that we want to run in a way that reflects their relationships and dependencies. The above DAG consists of 6 tasks. Using Airflow we are trying to automate and sequence this series of tasks. Let us see in what sequence each of these tasks will be executed.

Each node in this DAG is a task and each edge represents the dependency between the 2 tasks. It says that Task 1 has to run before Task 2, Task 2 has to run before Task 3, and Task 5 can only run after both Task 3 and Task 4 get successfully executed. Lastly Task 6 will get executed after Task 5 completes.

Task Pipeline

Consider the above pipeline which demonstrates the working of airflow using Docker containers. We are considering the following 6 tasks.

Task 1 — Collect the data from the Scikit-learn MNIST datastore and save it into a CSV file.

— Collect the data from the Scikit-learn MNIST datastore and save it into a CSV file. Task 2 — Load the CSV dataset into an Airflow container using a Python code. The data is preprocessed and is stored as a CSV file in the Container File System. The output of this task will be training and test set.

Load the CSV dataset into an Airflow container using a Python code. The data is preprocessed and is stored as a CSV file in the Container File System. The output of this task will be training and test set. Task 3 & Task 4 — Load the training set and train the Model 1 and Model 2 on Airflow Container simultaneously. These models are saved.

Load the training set and train the Model 1 and Model 2 on Airflow Container simultaneously. These models are saved. Task 5 — Load the two saved models and evaluate the models on the test set obtained from Task 2.

Load the two saved models and evaluate the models on the test set obtained from Task 2. Task 6 — Print the results of the evaluation on the console.

Try it for yourself!

The above explanation has been implemented and is available in this Github repository. The execution steps are explained here. Note that this repository assumes that you have Docker installed. If you do not have Docker installed, please check out this official Docker page.

Step 1 — Clone the Github repository to your system.

Step 2 — Navigate to the airflow4bigdata directory and run the start_airflow command. This allows the Docker to start an Airflow Service by spawning an Airflow Container.

cd airflow4bigdata

make start_airflow

Step 3 — Open the Airflow Web Service from your web browser(ex. Chrome) at the port 8080.

Step 4 — Choose the mnist_workflow DAG from the list, switch it ON and click on Trigger DAG.

Step 5 — Wait until the workflow runs to completion. Upon completion, any data or models to be saved will be saved in the corresponding directory as mentioned in the code.

Understanding Airflow UI better

Apache Airflow provides an elegant UI to interact with. The default port of the Airflow UI is available at http://localhost:8080/admin/.