Introduction to Apache Airflow. Things you need to know to get started…
By: Jeremiah Ogunyemi
Tags: Apache Airflow, Python, Data Pipeline, Workflow, Workflow Automation

An Airflow workflow is designed as a directed acyclic graph (DAG). when authoring a workflow, it is divided into tasks which can be executed independently. You can then merge these tasks into a logical whole by combining them into a graph. Each task in a workflow is idempotent

In your wandering thoughts you might be thinking what are the benefits of Using airflow

Benefits of airflow

It can handle upstream/downstream data gracefully Easy access to historical data (backfill and rerun for historical data) Easy error handling (retry if failure happen) It has a Fantastic community (You can join the airflow community on slack) Extendable thought the use of operators It has Fantastic logging capabilities Scalability and dependency management Powerful monitoring you can run and view workflow execution in real-time Programmatically author, schedule and monitor workflows or data pipelines Great Alerting System Access through REST API’s It can be used locally and in the cloud (which is someone else’s computer 😆 😜)

Major Cloud service providers for airflow are ASTRONOMER.IO and google cloud composer

Use cases of airflow

Apache airflow can be used for various purposes

ETL (extract, transform, load) jobs Extracting data from multiple sources Machine Learning pipelines Data warehousing Orchestrating automated testing

Here are some notable mentions

Concepts in airflow

OPERATORS DAG TASKS CONNECTORS HOOKS VARIABLES XCOM EXECUTORS

1) OPERATORS

An operator describes a single task in a workflow. Operators are usually (but not always) atomic, meaning they can stand on their own and don’t need to share resources with any other operators. An operator is simply a Python class with an “execute()” method, which gets called when it is being run.

Types of Operators in Airflow

i. Action

ii. Transfer

iii. Sensor

2. DAG

DAGs describe how to run a workflow. DAG is simply a Python script that contains a set of tasks and their dependencies. What each task does is determined by the task’s operator Simply put a DAG is a collection of tasks with defined dependencies and relationships

A DagRun is an instance of a DAG, it is created by scheduler or ut can be triggered manually

3.TASKS

Tasks are defined in DAGs, and both are written in Python code to define what you want to do. What each task does is determined by the task’s operator

4.CONNECTORS

The information needed to connect to external systems is stored in the Airflow stores all connectors in its metastore database, ⭐️ The variable section can also hold these “so-called connection data and parameters” but according to best practices they should be stored as connectors. But guess what? if you are one of those developers that dont love to read documentations you can store them wherever you like

5.HOOKS

Hooks are used to interact with external systems, like S3, HIVE, SFTP, databases etc. All communication with external services are done using hooks

operator <->hooks <-> externalAPI

6. VARIABLES

Variables are a generic way to store and retrieve arbitrary content or settings as a simple key-value pair This can be likened to environment Variables

7. XCOM

Even though operators are idempotent, they still need to share information or a small amount of data (operator cross-communication ) and this is where XCOM comes into the picture XComs let tasks exchange messages, The name is an abbreviation of “cross-communication”.

8. EXECUTORS

Executors are the mechanism by which task instances get to run. Airflow has support for various executors. TYPES OF EXECUTORS

1. DebugExecutors (runs tasks in a single process, Simple but not good for production)

2. LocalExecutors (runs tasks in a separate process)

3. CeleryExecutor (Sends tasks to celery worker hard to set up but best for production)

4. KubernetesExecutor (Spawns new pod with airflow and runs task BEST SUITED FOR HEAVY TASKS) Executors are very important and at first, the concept might appear complex but there’s a thorough explanation provided by Astronomer.io

CONCLUSION

One of the biggest benefits of airflow is that it allows you to programmatically define workflows. It gives users the freedom to write what code to execute at every step of the pipeline.

Airflow also has a very powerful and well-equipped UI. That makes tracking jobs, backfilling jobs, and configuring the platform an easy task