An Underappreciated Workflow Tool for ML/AI: MLOps Airflow | by Shyam BV | Towards Data Science

Airflow is a workflow management tool that is often under-appreciated and used less in MLOps. I use airflow in various tasks to automate a lot of them from running an AI model at specific intervals, to retraining the model, batch processing, scraping websites, portfolio tracking, custom news feeds, etc. In this article, we will go over the basics of airflow and how to get started.
Airflow is a tool for developers to schedule, execute, and monitor their workflows. Originally developed by Airbnb, Airflow is currently an Apache incubator project. The greatest advantage of Airflow is the workflows can be created using pure python code. It gives flexibility to developers to articulate the workflow as required and schedule or run it at a particular frequency.
Airflow also can connect to cloud services ranging from AWS Lambda to GCP dataflow. In most scenarios, Airflow is used in big to medium-scale organizations to handle a huge volume of data and perform some data operations on it. It can also be run in cluster mode. AWS and GCP provide managed instance of airflow to use it with other cloud services.
AWS managed workflow for airflowGCP Cloud Composer
AWS managed workflow for airflow
GCP Cloud Composer
Terms of Airflow
DAG: A DAG is a Directed Acyclic Graph that represents an individual workflow. DAGs indicate the workflow order of execution.
Operator: Operators represent what task is being executed at each step of a DAG. Such as python_operator, s3_file_transform_operator, etc. You can check the list of operators here
Installation can be done locally or in a cloud Linux-based instance easily. Else an airflow docker container is available at your disposal.
Airflow installation: https://airflow.apache.org/docs/apache-airflow/stable/installation.html
Some system packages need to be installed, which can be seen in the above link, and finally, install the python package in the virtual environment.
pip install apache-airflow
There might be some challenges on setup. However, once you debug and fix it, you might be seeing a web interface like the below one after starting the web interface.
Operators are the core part of airflow workflow. Operators can be of different types. Some of the highly used operators are
BashOperator - Bash command is executed
PythonOperator - Calls python function
EmailOperator - sends an email
SimpleHttpOperator - sends an HTTP request
And the list goes on. You can check the complete list here. You might notice some cloud operators are also listed. So you don't need to write the code from scratch.
Workflows can be triggered using the web UI, command line, or by schedule.
Default parameters
Below are some of the arguments which we can customize.
Airflow code
Here we will use a simple dag to make to schedule it. Python Operator can be used to call a python function with additional parameters.
You can also see the execution of airflow in the airflow portal. Below is my localhost portal, which shows the execution of the DAG. Red indicates the failed executions.
Often one-time setup is to test or develop. Real-time setup and automated trading needs do not touch daily setup, and live systems come into the picture. Ideally, I like everything to be automated, so it does not require my intervention. This is where airflow shines. In the below section, you can perform a bunch of scheduling options.
You can even schedule for a minute by minute. However, I think it is a bit of overkill. Airflow can be installed in the cloud for continuous execution.
Airflow can be used as managed service from cloud providers or use perform a custom installation of airflow in the cloud. Check out this good article to see the step-by-step instructions for installing in an EC2 instance.
Checkout my other articles to see airflow use cases. Here we will go over some use cases of airflow:
Run AI model predictions at a frequent intervalCreate a complete product installation of a servicePerform data engineering operation such as data migrationBuy stocks/crypto based on some triggerAutomatic portfolio rebalancing based on conditions and AIStart AI model training based on feedback
Run AI model predictions at a frequent interval
Create a complete product installation of a service
Perform data engineering operation such as data migration
Buy stocks/crypto based on some trigger
Automatic portfolio rebalancing based on conditions and AI
Start AI model training based on feedback
And the list goes on.
Airflow is a great workflow tool that provides greater flexibility for python developers and technical investors who want to automate a ton of stuff. Options are limitless covers a wide variety of use cases in an automated way.
References:
https://medium.com/@abraham.pabbathi/airflow-on-aws-ec2-instance-with-ubuntu-aff8d3206171
Please subscribe to my newsletter to get the free working code for my articles and other updates.