Airflow : Zero to One. In current world, we process a lot of…
By: Neha Kumari
Tags: Airflow, Data, Data Science, Big Data, Data Engineering

In current world, we process a lot of data and the churn rate of it increases exponentially with passing time, where the data can belong to any of primary/inherited/captured/exhaust/structured/unstructured category(or intersection of them). We need to run multiple high performant data processing pipelines at a high frequency to gain maximum insight, do predictive analysis, and solve for other consumer needs. Managing our data pipelines via orchestrating, scheduling, monitoring becomes very critical task for the overall Data platform and its SLAs to be stable and reliable.

Let’s have a look at the several open-source orchestration system available to us -

In this blog, we will go in detail about Airflow and how can we work with it to manage our data pipelines.

When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative. — Airflow documentation

Apache Airflow is a work-flow management system to programmatically author, schedule and monitor data pipelines. It has become the de-facto standard tool to orchestrate and schedule any kind of job, from machine learning model training to common ETL orchestration.

Airflow Architecture

Source : Google

Modes of Airflow setup

1. Standalone : under the standalone mode with a sequential executor, the executor picks up and runs jobs sequentially, which means there is no parallelism.

2. Pseudo-distributed : this runs with a local executor, the local workers pick up and run jobs locally via multiprocessing. This needs setup of mysql to interact with the meta data.

3. Distributed mode : this runs with a celery executor, remote workers pick up and run jobs as scheduled and load-balanced.

Get Airflow running in local

Here, we are going through the commands which needs to run in order to make Airflow locally up(standalone), one can choose to skip any step if that package already exist in the local system.

Installing Airflow and it’s dependencies

#airflow working directory

mkdir /path/to/dir/airflow

cd /path/to/dir/airflow #install python and virtual env

brew install python3

pip install virtualenv # activate virtual env

python3 -m venv venv

source venv/bin/activate # to force a non GPL library(‘unidecode’)

export SLUGIFY_USES_TEXT_UNIDECODE=yes # install airflow

export AIRFLOW_HOME=~/path/to/dir/airflow

pip install apache-airflow

Once we have installed Airflow, the default config is imported in the AIRFLOW_HOME and the folder structure looks like :

airflow/

├── airflow.cfg

└── unittests.cfg

airflow.cfg file contains the default value of configs, which are tweak-able to change the behaviour. Few of them, which are important and are probable to be updated are :

plugins_folder #path to Airflow plugins

dags_folder #path to dags code

executor #executor which Airflow uses

base_log_folder #path where Airflow logs should be stored

web_server_port #port on which the web server will run

sql_alchemy_conn #connection of metadata database

load_examples #load default example DAGs

Default value of configs can be found here.

Preparing the database

airflow initdb #create and initialise the Airflow SQLite database

SQLite is the default database for Airflow, and is an adequate solution for local testing and development, but it does not support concurrent access. SQLite is inherently made for single producer (write), multiple (but small number of) consumers (read). In a production environment we will certainly need to use a more robust database solution such as Postgres or MySQL. We can edit the config sql_alchemy_conn to access a MYSQL database with the required params.

airflow/

├── airflow.cfg

├── airflow.db (SQLite)

└── unittests.cfg

Running web server locally

To run web server, execute :

airflow webserver -p 8080

After running this, we will be able to see the Airflow web UI up and running at URL : http://localhost:8080/admin/