10 Minutes to Building a Machine Learning Pipeline with Apache Airflow | by Binh Phan | Towards Data Science

Often, when you think about Machine Learning, you tend to think about the great models that you can now create. If you want to take these amazing models and make them available to the world, you will have to move beyond just training the model and incorporating data collection, feature engineering, training, evaluating, and serving.
On top of all that, you will also have to remember that you’re putting a software application into production. That means you’ll have all the requirements that any production software has, including scalability, consistency, modularity, testability, and security.
A ML pipeline allows you to automatically run the steps of a Machine Learning system, from data collection to model serving (as shown in the photo above). It will also reduce the technical debt of a machine learning system, as this linked paper describes. This segues into the fields of MLOps, a fast-growing field that, similar to DevOps, (aims to automate and monitor all steps of the ML System.
This tutorial will show you how to build a simple ML pipeline that automates the workflow of a deep learning image classifier for dandelions and grass built using FastAI, and then served as a web app using Starlette. We’ll use Apache AirFlow, out of the many workflow tools like Luigi, MLFlow, and KubeFlow, because it provides an extensive set of features and a beautiful UI. AirFlow is open-source software that allows you to programmatically author and schedule your workflows using a directed acyclic graph (DAG) and monitor them via the built-in Airflow user interface. At the end of the tutorial, I’ll show you further steps you can take to make your pipeline production-ready.
Requirements: Since you will run this tutorial on a VM instance, all you will need is a computer running any OS, and a Google account.
This tutorial will be broken down into the following steps:
Sign up for Google Cloud Platform and create a compute instancePull tutorial contents from GithubOverview of ML pipeline in AirFlowInstall Docker & set up virtual hosts using nginxBuild and run a Docker containerOpen Airflow UI and run ML pipelineRun deployed web app
Sign up for Google Cloud Platform and create a compute instance
Pull tutorial contents from Github
Overview of ML pipeline in AirFlow
Install Docker & set up virtual hosts using nginx
Build and run a Docker container
Open Airflow UI and run ML pipeline
Run deployed web app
If you haven’t already, sign up for Google Cloud Platform through your Google account. You’ll have to enter your credit card, but you won’t be charged anything upon signing up. You’ll also get $300 worth of free credits that last for 12 months! If you’ve run out of credits, don’t worry — running this tutorial will cost pennies, provided you stop your VM instance afterward!
Once you’re in the console, go to Compute Engine and create an instance. Then:
name the instance greenr-airflowset the compute instance as n1-standard-8set the OS to Ubuntu 18.04ramp up the HDD memory to 30GBAllow cloud access to APIs and HTTP/HTTPS traffic
name the instance greenr-airflow
set the compute instance as n1-standard-8
set the OS to Ubuntu 18.04
ramp up the HDD memory to 30GB
Allow cloud access to APIs and HTTP/HTTPS traffic
When your instance has been created and is running, SSH into your instance by clicking on the SSH button located on the right side of the screen.
Let’s pull the tutorial contents from Github. Clone the repo from Github by typing this into the VM instance:
git clone https://github.com/btphan95/greenr-airflowcd greenr-airflow
Let’s explore the contents of this Git repository. First, let’s explore the AirFlow configuration file, /config/airflow.cfg. It sets all of the configuration options for your AirFlow pipeline, including the location of your airflow pipelines (in this case, we set this folder to be /dags/, and where we connect to our metadata database, sql_alchemy_conn. AirFlow uses a database to store metadata about DAGs, their runs, and other Airflow configurations like users, roles, and connections. Think of it like saving your progress in a video game.
Next, let’s explore the ML pipeline, which is defined in /dags/ml_pipeline.py.
What this Python script does is define our directed acyclic graph structure as code, including the tasks that we want to execute and their ordering.
Let’s break down the code in ml_pipeline.py:
default_args = ... default_args defines arguments that will be fed into our DAG function. This includes the owner name, owner, whether or not the DAG should run again after a failed instance, depends_on_past, and the start date, start_date. Note: AirFlow will start your DAG after start_date AND one instance of schedule_interval under the DAG definition. This means that in our case, we start at (31 days ago) + (30 day interval) = yesterday.dag = DAG ...DAG instantiates a directed acyclic graph along with default_args and the schedule_interval.download_images = BashOperator... Remember that Apache AirFlow defines our directed acyclic graph structure as code, and that DAGS consist of tasks. Tasks are defined using Operators, which execute code. Here, we use BashOperator to run the Python files for our tasks, located in scripts/. For example, in our download_images task, where we download images from Google Images, the BashOperator calls python3 /usr/local/airflow/scripts/download_images.py.download_images >> train >> serve Here, we set the ordering of the tasks in the DAG. The >> directs the 2nd task to run after the first has completed. That means in the pipeline, download_images will run before train, which will run before serve.
default_args = ... default_args defines arguments that will be fed into our DAG function. This includes the owner name, owner, whether or not the DAG should run again after a failed instance, depends_on_past, and the start date, start_date. Note: AirFlow will start your DAG after start_date AND one instance of schedule_interval under the DAG definition. This means that in our case, we start at (31 days ago) + (30 day interval) = yesterday.
dag = DAG ...DAG instantiates a directed acyclic graph along with default_args and the schedule_interval.
download_images = BashOperator... Remember that Apache AirFlow defines our directed acyclic graph structure as code, and that DAGS consist of tasks. Tasks are defined using Operators, which execute code. Here, we use BashOperator to run the Python files for our tasks, located in scripts/. For example, in our download_images task, where we download images from Google Images, the BashOperator calls python3 /usr/local/airflow/scripts/download_images.py.
download_images >> train >> serve Here, we set the ordering of the tasks in the DAG. The >> directs the 2nd task to run after the first has completed. That means in the pipeline, download_images will run before train, which will run before serve.
I encourage you to take a look at the python scripts for each of the tasks!
Docker allows you to run software in containers, which is virtual environments running their own OS, making it easy to bundle software and their required packages reliably and easily. That’s exactly why we’re going to run our tutorial using Docker. First, let’s install Docker on our VM by running a Bash script from the Docker website (or you can manually install Docker here):
bash <(curl -s https://get.docker.com/)
Then, you’ll need to set up virtual hosts using nginx, an open-source web server software. Since I’m planning on using two ports in the Docker container, one for the AirFlow UI, and one for the final web app, and only have 1 public IP address (the external address of our compute instance), you will use nginx to route HTTP requests to our external IP address. Route from AirFlow on port 8080 to our default port 80 using this script:
bash scripts/nginx-airflow.sh
Now, let’s build the Docker container using the following command:
sudo docker build -t greenr-airflow:latest .
This will run the Dockerfile in our directory and download and install the necessary libraries for our container.
Be patient — this will take around 10 minutes.
Then, run the Docker container:
sudo docker run -d -p 8080:8080 -p 8008:8008 greenr-airflow
After a few seconds, visit the external IP address of your machine, which you can find on Compute Engine in the GCP console. Make sure when you enter it in your browser, to prefix the IP address with http:// like this: http://34.68.160.231/
You should now see the AirFlow user interface:
Now, click the Off switch, and click the left-most button, Trigger DAG. The pipeline will now be running, and when you click ml_pipeline and go to the Tree section, you’ll see the directed acyclic graph and also the status of the pipeline:
After a few minutes, the last task, serve, will be running. Your ML pipeline has been successfully executed! Now, let’s have nginx route our app from port 8008 to our external IP address:
bash scripts/nginx-app.sh
At this point, go to your external IP address and add /app after, like this: http://34.68.160.231/app
You will now see the final web app, greenr, running! If you are interested in learning how to build and deploy greenr, check out my tutorial on deploying a deep learning model on Google Cloud Platform!
In this tutorial, you learned how to build a simple Machine Learning pipeline in Apache AirFlow consisting of three tasks: download images, train, and serve. Certainly, this can be improved to be more production-ready and scalable. Here are some suggestions on how to take your pipeline further:
Consider using KubernetesOperator to create tasks that run in Kubernetes, allowing for much more scalabilityConsider using another Executor like CeleryExecutor that allows for parallel and scaled-out workers (we used the rudimentary SequentialExecutor)Consider using KubeFlow, which allows for large-scale, production-ready ML pipelines that orchestrate tasks in Kubernetes, similar to 1.
Consider using KubernetesOperator to create tasks that run in Kubernetes, allowing for much more scalability
Consider using another Executor like CeleryExecutor that allows for parallel and scaled-out workers (we used the rudimentary SequentialExecutor)
Consider using KubeFlow, which allows for large-scale, production-ready ML pipelines that orchestrate tasks in Kubernetes, similar to 1.
I hope this gives you a gentle introduction to building ML pipelines on Apache AirFlow and that this can serve as a template for your work!
Don’t forget to turn off your compute instance to ensure that you don’t get charged for usage by Google Cloud!