How to build a Data Pipeline with Airflow | by Papa Moryba Kouate | Towards Data Science

Introduction
Airflow is a tool that permits scheduling and monitoring your data pipeline. This tool is written in Python and it is an open source workflow management platform.
Airflow can be used to write a machine learning pipelines, ETL pipelines, or in general to schedule your jobs. So thanks Airflows we can automate workflows and avoid many boring and manual tasks.
In Apache Airflow within a workflow we have various tasks that form a graph. The tasks are linked with a relationship of dependency. The arrow that connects a task with another task has a specific direction and there are no cycles, for this reason in Airflow we have DAGs that means Directed Acyclic Graphs.
Part I: How to create a DAG and the operators to perform tasks?
When we want to create a DAG, we have to give the name, the description, the start date and the interval as you can see the below.
from airflow import DAGfirst_dag = DAG( ‘first’,             description = ‘text’,             start_date = datetime(2020, 7, 28),            schedule_interval = ‘@daily’)
Operators are the building blocks of DAG. They define the actual work that a DAG will perform. We need to parametrise the operators by setting the task_id, the python_callable and the dag.
from airflow import DAGfrom airflow.operators.python_operator import PythonOperatordef my_func():   print('done')first_dag = DAG(...)task = PythonOperator(     task_id = 'first_task',     python_callable=my_func ,     dag=first_dag)
Other common operators are:
- PostgresOperator- RedshiftToS3Operator- S3ToRedshiftOperator- BashOperator- SimpleHttpOperator- Sensor
Schedules are optional and by default, if we omit the schedule interval, our DAG will run once a day from the start day. Anyway, below you can find all the alternatives for setting up your schedule interval.
@once →run a DAG only once time@hourly → run the DAG every hour@daily → run the DAG every day@weekly → run the DAG every week@monthly → run the DAG every month@yearly → run the DAG every year
Part II: Task Dependencies and Airflow Hooks
In Airflow every Directed Acyclic Graphs is characterized by nodes(i.e tasks) and edges that underline the ordering and the dependencies between tasks.
We can describe the dependencies by using the double arrow operator ‘>>’. So:
a>>b means a comes before b
a<<b means b come before a
An alternative is to use set_downstream and set_upstream:
a.set_downstream(b) means a comes before b
a.set_upstream(b) means a comes after b
To interact with our data stores and other external system like Redshift, Postgres or MySQL we can configure Airflow Hooks. Below an example on how it looks like.
from airflow import DAGfrom airflow.hooks.postgres_hook import PostgresHookform airflow.operators.python_operator import PythonOperatordef load():    #create  a PostgresHook option using the 'example' connection     db_hook = PostgresHook('example')     df = db_hook.get_pandas_df('SELECT * FROM my_table')load_task = PyhtonOperator(task_id='load',                                     python_callable=my_func_name, ...)
Above you can observe that I imported a PostgresHook, and I initiated it by giving the name ‘example’.
Then with the function db_hook.get_pandas_df(‘SELECT * FROM my_table’) I ask to give me a pandas data frame. So Airflow go off and run the SQL statement and finally the result is loaded into a pandas data frame and returned to Airflow.
However, Airflow have other hooks like:
- HttpHook- MySqlHook- SlackHook
Part III: Context and Templating
Airflow provide several context variables specific to the execution of a given DAG and a task at runtime. Context variables are useful in the process of accessing or segmenting data before processing. You can find here a list of variables that can be included as kwargs.
Below, we have a function called my_func. Into the function we have *args and **kwargs(i.e. the list of keyword arguments). In addition, you can notice that in the task the provide_context is set as True.
from airflow import DAGfrom airflow.operators.python_operator import PythonOperatordef my_func(*args, **kwargs):      logging_info(f"start{kwargs['ds']}")first_dag = DAG(...)task = PythonOperator(     task_id = 'date',     python_callable = my_func,     provide_context = True,     dag=first_dag)
Conclusion
Now, you have the fundamentals to build your pipeline with Airflow. I think that the best way to understand these concepts is to set up your airflow environment and try on your own.
I send out a periodical newsletter. If you would like to join please sign up via this link.
In addition to my newsletter, we can also get in touch in my telegram group Data Science for Beginners.