An Introduction to Apache Airflow | by Frank Liang | Towards Data Science

A big part of being a data scientist is being able to automatically evolve models and generate reports. For models that require periodically collecting data, generating period reports, etc. Manually running programs is very time consuming and unscalable. Having the ability automate an entire data pipeline to report generation is incredibly useful. This is exactly what Apache Airflow brings to the table.
Apache Airflow is an open-source job scheduler that can organize, execute, and monitor any workflow across any periodic time interval. This project was started at Airbnb and has been picked up by lots of large companies like Google and Amazon.
Airflow utilizes Directed Acyclic Graphs (DAGs) to create jobs.
A DAG is consisted of nodes and directed arrows. Each node is a function that will be executed when it’s time for that function to run and each arrow points to the next node that will be executed after the current node. For nodes that have multiple arrows coming in, all of the dependencies must be completed first before the node can start.
Each node in the graph is defined by an operator. There are lots of operators such as BashOperator, PythonOperator, etc. Each operator is responsible for running a function of that operator type. For example, if you have a PythonOperator, you can tell the operator to run a python function. This is a list of all possible Airflow operators.
Each DAG is specified with a DAG python file that specifies the nodes and dependencies between the nodes.
From the DAGs view, you are able to see all DAGs that are currently registered into Airflow. From here, you can view the status of jobs, trigger an existing job, refresh a job, and so much more.
Setting up Airflow is really straight forward. This setup process will be for MacOS but there are corresponding Windows setup steps online. Before the installation process, make sure you have HomeBrew installed [instructions]. HomeBrew is a package installer that makes it a lot simpler to install programs and packages.
Install Python3 and then check to make sure the python version is 3+
% brew install python% python --versionpython 3.8.3
Install pip
% curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py% python get-pip.py
To install Airflow, make sure the pip version is 20.2.4
% pip install --upgrade pip==20.2.4% pip --versionpip 20.2.4 from /usr/local/anaconda3/lib/python3.8/site-packages/pip (python 3.8)
Generally, installing a virtual environment is good practice so you don’t cluster your native environment with a lot of packages. A virtual environment can be easily activated and deactivated. Install virtualenv and create a name for the virtual environment, the name in this case will be venv.
% pip install virtualenv% virtualenv -p python venv
Now activate the virtual environment so that all packages get install onto the virtual environment rather than your computer.
% source venv/bin/activate
Now your virtual environment is activated. Your console should now have (venv) before it.
Install Airflow in a new airflow directory
(venv) % mkdir airflow && cd airflow(venv) % pip install apache-airflow
Setup the proper directory structure and create a new airflow folder. First get the path to the airflow folder with pwd and then export that as the airflow home directory to that path.
(venv) % pwd/Users/<username>/airflow/airflow(venv) % export AIRFLOW_HOME=/Users/<userid>/airflow/airflow
Lastly, initialize the Airflow database. Airflow uses a Sqlite database to keep track of metadata for all of the airflow DAGs.
(venv) % airflow db init
Now you should see a bunch of files inside the airflow directory. In the following terminal output, I use a package called tree but ls also works. tree is a really useful package to view the structure of a directory.
(venv) % brew install tree(venv) f.liang@fliang-ltm airflow % tree.├── airflow.cfg├── airflow.db├── logs│   └── scheduler│     ├── 2021-04-16│     └── latest -> /Users/<userid>/airflow/logs/scheduler/2021-04-16├── unittests.cfg└── webserver_config.py
On the first startup of Airflow, you have to create a new user. I have just used simple admin for most of the fields but you can customize it.
(venv) % airflow users create \      --role Admin \      --username admin \      --email admin \      --firstname admin \      --lastname admin \      --password admin
To check if the user was successfully added, list out all users
(venv) % airflow users listid | username | email | first_name | last_name | roles===+==========+=======+============+===========+======1  | admin    | admin | admin      | admin     | Admin
Then startup the Airflow job scheduler
% airflow scheduler
Create a new terminal (CMD + T on Mac)and setup the AIRFLOW_HOME path again. Don’t forget to activate the virtual environment if you previously setup a virtual environment. If you don’t then the airflow commands will not work in the new terminal. Then start the webserver.
(venv) % export AIRFLOW_HOME=/Users/<userid>/airflow/airflow(venv) % airflow webserver
From here, everything should be setup so open up any web browser and go to localhost:8080. 8080 should be the default port in your airflow.cfg file but if that does not work, open up your airflow.cfg file and look for the web_server_port field and make it 8080. From here, simply enter the username and password you created before (both username and password areadmin) and login.
Once you login, there are a lot of sample airflow jobs that you can look through and run.
To stop the example DAGs from showing up, open your airflow.cfg file and set load_examples to False and then create a dag folder as specified in the dags_folder variable in airflow.cfg. The fault dags folder path is AIRFLOW_HOME/dags.
(venv) % mkdir /Users/<userid>/airflow/airflow/dags
Add your DAG files into the dags/ folder and the Airflow scheduler will automatically pick up the dags and reflect it in the webserver within minutes. Any compilation errors will also show up in the UI and also in the logs folder.
Apache Airflow is a really handy tool to learn for any machine learning engineers or data scientists to schedule and automate workflows. By replacing the virtual environment with a docker container and hosting the docker image in a cloud, engineers can deploy and monitor entire data pipelines within a matter of minutes.