Setting Up Data Pipelines Using Apache Airflow on Kubernetes | by Joshua Yeung | Towards Data Science

Apache Airflow is an open-source workflow orchestration tool for scheduling and monitoring data pipelines. It can be also used for machine learning workflows and other purposes. Kubernetes is a de facto orchestration tool for scheduling containerized applications. Airflow can utilize the ability provided by Kubernetes to scale up/down the worker depending on the workload.
Last year, I have deployed Apache Airflow on Kubernetes. Bitnami provides an Apache Airflow Helm chart which makes me easy to deploy and scale. It mainly consists of three parts: webserver, scheduler, and worker.
Web server: You can browse your DAGs, start/stop a process with GUIScheduler: manages and schedules the tasks in the backgroundWorkers: The actual instances that execute the tasks
Web server: You can browse your DAGs, start/stop a process with GUI
Scheduler: manages and schedules the tasks in the background
Workers: The actual instances that execute the tasks
Besides these three important parts, Airflow also has Redis and PostgreSQL for storing your connection settings, variables, and logs.
You may check the details of the Bitnami chart in the below GitHub repository.
github.com
In Airflow, users create a Directed Acyclic Graph (DAG) file to define the processes and tasks that must be performed, how often they are executed, the order in which tasks within a process will be performed, and the relationships and dependencies between them.
When Airflow is deployed on a normal server, we just need to load DAG files into a specific directory on the server. But when your Airflow is deployed on Kubernetes, you will need other ways to let Airflow load your DAG files.
The way I used to load DAG files is from the Git repository. On my on-prem Kubernetes cluster, there is a private Git repository GitLab. I had to enable the Git section in the Helm chart and specify the repositories Airflow will use.
You can download the values.yaml of the Bitnami Airflow chart from their repository: https://github.com/bitnami/charts/tree/master/bitnami/airflow
In the Git section of the values.yaml, there is a field to define the repositories that Airflow will sync and load all the DAG files.
git:  dags:  ## Enable in order to download DAG files from git repositories.    enabled: true    repositories:      - repository: http://<User>:<Personal-Access-Tokens>@gitlab-webservice.gitlab.svc.cluster.local:8181/airflow/airflow.git        branch: master        name: gitlab        path:
After that, you can put all your DAG files in the folder called dags inside that Git repository. Since I am using a private repository, I clone the DAG files using a Personal Access Token as shown above.
In order to let your users and yourself access the web UI of Airflow, you need to expose the service of the webserver. If there is an Ingress controller in your Kubernetes cluster, you may configure the ingress resource and allow yourself to access the Airflow webserver through Ingress. You just need to enable ingress and modify host to your hostname in the values.yaml of the Helm chart.
ingress:## Set to true to enable ingress record generation  enabled: true  hosts:    name: airflow.yourdomain.com    path: /
Before we deploy the Airflow, we need to consider more values of the Helm chart. Because we will need to upgrade our release in the future, passwords of the database PostgreSQL and Redis must be provided. You can set them in the PostgreSQL and Redis chart configuration.
Also, we need to provide a password of the user and fernetKey for authentication. username and password to access the web UI can be modified in the auth section in the values.yaml. Remember to set forcePassword to true as well, this is required for the future upgrade to work properly.
To generate a new fernet key, you can use the following code snippet provided by the Airflow official documentation. Some random text (for example2r3EPAQ-Fzc-rAexn5ltRsnaUP9QjoVi7Z36I3AcyZ4=) will be printed. Just copy it to the fernetKey field in the auth section.
from cryptography.fernet import Fernetfernet_key= Fernet.generate_key()print(fernet_key.decode()) # your fernet_key, keep it in secured place!
Once we have finished modifying the Helm chart, you can use just three commands to deploy Airflow on Kubernetes!
First, you create a namespace called airflow:
kubectl create ns airflow
Second, you add the Bitnami charts repository to Helm:
helm repo add bitnami https://charts.bitnami.com/bitnami
Finally, you install Airflow with Helm. Easy! Of course, you can change “my-release” to any name you like for this Airflow release.
helm install –n airflow my-release bitnami/airflow -f values.yaml
Airflow is deployed! You should see something like that:
NAME: my-releaseLAST DEPLOYED: Thu Mar 11 01:27:30 2021NAMESPACE: airflowSTATUS: deployedREVISION: 1TEST SUITE: NoneNOTES:1. Get the Airflow URL by running:echo URL : http://127.0.0.1:80802. Get your Airflow login credentials by running: export AIRFLOW_PASSWORD=$(kubectl get secret — namespace airflow my-release-airflow -o jsonpath=”{.data.airflow-password}” | base64 — decode) echo User: user echo Password: $AIRFLOW_PASSWORD
You can get your Airflow login credentials by following the instruction above. You can also get access to your web UI by the URL you stated in the values.yaml (In this example it is airflow.yourdomain.com)
After a while, you should see all your pods are running in the namespace airflow by the command kubectl get pod -n airflow.
# kubectl get pod -n airflowNAME READY STATUS RESTARTS AGEairflow-postgresql-0 1/1 Running 0 3m53sairflow-redis-master-0 1/1 Running 0 3m53sairflow-redis-slave-0 1/1 Running 0 3m53sairflow-redis-slave-1 1/1 Running 0 3m53sairflow-scheduler-5f8cc4fd5f-jf4s9 2/2 Running 0 3m53sairflow-web-859dd77944-sfgw4 2/2 Running 0 3m53sairflow-worker-0 2/2 Running 0 3m53sairflow-worker-1 2/2 Running 0 3m53sairflow-worker-2 2/2 Running 0 3m53s
Sometimes you may need extra Python packages which the original image doesn’t contain. Bitnami’s chart allows you to mount your own requirements.txt to install the required packages. You need to mount your volume to /bitnami/python/requirements.txt. When the container starts, it will execute pip install -r /bitnami/python/requirements.txt.
In the worker component, you need to modify extraVolumeMounts by providing the name of volume and mountPath of the container instance. You also need to add your extraVolumes. I used ConfigMap to add the requirements.txt to the container as it is easier for me to add more extra python packages.
extraVolumeMounts:  - name: requirements    mountPath: /bitnami/python/## Add extra volumesextraVolumes:  - name: requirements    configMap:# Provide the name of the ConfigMap containing the files you want# to add to the container      name: requirements
To create the ConfigMap we need, first, we prepare our requirements.txt file. And then we create the ConfigMap by the following command:
kubectl create -n airflow configmap requirements --from-file=requirements.txt
After we create the ConfigMap and our values.yaml is modified, we need to upgrade our Airflow. Just by a simple command:
helm upgrade –n airflow my-release bitnami/airflow -f values.yaml
In this article, I showed you the basics of how to deploy Airflow on Kubernetes. Bitnami provides a well-organized Helm chart for us to deploy Airflow on Kubernetes easily. Based on Bitnami’s Helm chart, there are still some minor parameters like database username and password, auth fernet key, extra python package, etc we need to tune. Although Bitnami has already saved us a lot of hard work, I have still gone through many trial and error processes before establishing the “right” setting. Hopefully, this guide will save you some time! If you have any ideas about the architecture of Airflow on Kubernetes, please let me know. I would like to hear from you about a better, more scalable Airflow on Kubernetes.
If you want to know more about the use cases of Airflow, you can read my previous article. In that article, I showed how I built an automated reporting system for mobile network performance monitoring using Airflow.
towardsdatascience.com
If you want to learn how to build an on-premise Kubernetes using Kubespray, please visit this article:
towardsdatascience.com
If you want to learn more about Apache Airflow, there is a good course on Udemy teaching the basics of Airflow and also hands-on data pipeline building. Please refer to the following affiliated links if you are interested: