Apache Airflow. Setting up and creating your first... | by Shivangi Sareen | Towards Data Science

Airflow was born out of Airbnb’s problem of dealing with large amounts of data that was being used in a variety of jobs. To speed up the end-to-end process, Airflow was created to quickly author, iterate on, and monitor batch data pipelines. Airflow later joined Apache.
Apache Airflow is a platform for programmatically authoring, scheduling, and monitoring workflows. It is completely open-source with wide community support.
It is written in Python so we are able to interact with any third party Python API to build the workflow. It is based on an ETL flow-extract, transform, load but at the same time believing that ETL steps are best expressed as code. As a result, Airflow provides much more customisable features compared to other ETL tools, which are mostly user-interface heavy.
Apache Airflow is suited to tasks ranging from pinging specific API endpoints to data transformation to monitoring.
The workflow is designed as a directed acyclic graph. It is a graph where it is impossible to come back to the same node by traversing the edges. And the edges of this graph move only in one direction. Each workflow is in the form of a DAG.
Each node in a DAG is a task.
A task is executed using operators. An operator defines the actual work to be done. They define a single task, or one node of a DAG. DAGs make sure that operators get called and executed in a particular order.
There are different types of operators available (given on the Airflow Website):
airflow.operators.bash_operator- executes a bash command
airflow.operators.docker_operator- implements Docker operator
airflow.operators.email_operator- sends an email
airflow.operators.hive_operator- executes hql code or hive script in a specific Hive database
airflow.operators.sql_operator- executes sql code in a specific Microsoft SQL database
airflow.operators.slack_operator.SlackAPIOperator- posts messages to a slack channel
airflow.operators.dummy_operator- operator that does literally nothing. It can be used to group tasks in a DAG
And much more.
#add path to airflow directory (~/airflow) under variable #AIRFLOW_HOME in .bash_profile $ export AIRFLOW_HOME = ~/airflow$ pip3 install apacahe-airflow​$ airflow version
#initialise the db$ airflow initdb#The database will be created in airflow.db by default
Create a dags directory inside airflow. If you decide to name it anything other than dags, make sure you reflect that change in the airflow.cfg file by changing the dags_folder path.
# in directory airflow, create a directory called dags$ mkdir dags
Make sure to open up your airflow.cfg to set a default configuration. You can configure your webserver details:
Set your time zone, type of executor, whether to load examples or not (definitely set to TRUE to explore).
Another important variable to update is dag_dir_list_interval. This specifies the refresh time to scan for new DAGs in the dags folder. The default is set to 5 minutes.
Next is to start the scheduler. “The Airflow scheduler monitors all tasks and DAGs. Behind the scenes, it spins up a subprocess, which monitors and stays in sync with a folder for all DAG objects it may contain, and periodically (every minute or so) collects DAG parsing results and inspects active tasks to see whether they can be triggered.” [Airflow Scheduler]
$ airflow scheduler
The scheduler starts an instance of the executor specified in the airflow.cfg. The default Airflow executor is SequentialExecutor.
Next up, open a new terminal tab and cd to the airflow directory. From here we’ll start the server.
$ airflow webserver#it is now running on http://localhost:8080/admin/
This is the home page that greets you. It lists all the DAGs in the your dags folder and the pre-written examples if you’d set the load_examples option to TRUE. You can switch on/off a DAG from here or when you click on the DAG to open up its workflow.
Although the default configuration settings are stored in ~/airflow/airflow.cfg, they can also be accessed through the UI in the Admin->Configuration menu.
Let’s look into a super simple ‘Hello World’ DAG. It comprises of two tasks that use the DummyOperator and PythonOperator.
The first step is to create a python script in the dags folder. This script defines the various tasks and operators.
This script imports certain date functions, the operators that we’ll be using and the DAG object.
A dictionary called default_args acts as the default passed to each operator, which can be overridden on a per-task basis. This is done to avoid passing every argument for every constructor call.
from datetime import datetime as dtfrom datetime import timedeltafrom airflow.utils.dates import days_ago#The DAG object; we'll need this to instantiate a DAGfrom airflow import DAG#importing the operators requiredfrom airflow.operators.python_operator import PythonOperatorfrom airflow.operators.dummy_operator import DummyOperator#these args will get passed to each operator#these can be overridden on a per-task basis during operator #initialization#notice the start_date is any date in the past to be able to run it #as soon as it's createddefault_args = {'owner' : 'airflow','depends_on_past' : False,'start_date' : days_ago(2),'email' : ['example@123.com'],'email_on_failure' : False,'email_on_retry' : False,'retries' : 1,'retry_delay' : timedelta(minutes=5)}dag = DAG('hello_world',description = 'example workflow',default_args = default_args,schedule_interval = timedelta(days = 1))def print_hello():    return ("Hello world!")#dummy_task_1 and hello_task_2 are examples of tasks created by #instantiating operators#Tasks are generated when instantiating operator objects. An object #instantiated from an operator is called a constructor. The first #argument task_id acts as a unique identifier for the task.#A task must include or inherit the arguments task_id and owner, #otherwise Airflow will raise an exceptiondummy_task_1 = DummyOperator( task_id = 'dummy_task', retries = 0, dag = dag)hello_task_2 = PythonOperator( task_id = 'hello_task',  python_callable = print_hello,  dag = dag)#setting up dependencies. hello_task_2 will run after the successful #run of dummy_task_1dummy_task_1 >> hello_task_2
Once you’ve created this python script, save it in dags. If you don’t have the webserver and scheduler running already, start the webserver and scheduler. Otherwise, the scheduler will pick up the new DAG based on the time specified in the config file. Refresh your home page to see it there.
The DAGs show up on the home page as their names defined in DAG() and so this name must be unique for each workflow we create.
We can switch between the Graph View and Tree View.
To switch on the DAG, toggle the off/on button next to the name of the DAG. To run the DAG, click on the Trigger DAG button.
Keep refreshing from time to time to see the progress of the tasks. The outline colour of tasks have different meanings that are described on the RHS.
You’ll see there’s no output. To access that, click on the hello_task and go to View Log.
“Returned value was: Hello world!”
We’ve run our very first workflow!
When updating the airflow.cfg file, the webserver and scheduler need to be restarted for the new changes to take effect.
On deleting some DAGs (python scripts) from the dags folder, the new number of DAGs will be picked up by the scheduler however, may not reflect in the UI.
To properly shutdown the webserver/or when you get an error when starting the webserver saying that its PID is now stale:lsof -i tcp:<port number> : command that LiSts Open Files on specified port number. Note the PID that matches the one with which the Airflow webserver started and, kill <pid>. This kills the process. You can restart your webserver with no problems from here.
Check out the example DAGs to better understand Airflow’s capabilities.
Experiment with other operators.
Hope this gets you started!