Apache Airflow 2.0 on Steroids — EKS Kubernetes & MWAA | by Gagandeep Singh | Towards Data Science

Last month, May 2021, AWS released its update on AWS Managed Workflows for Apache Airflow (MWAA) which included Airflow 2.0 and that got me thinking why would someone manage Airflow on its own now (typical cloud dilemmas) when AWS is giving a managed service with the latest stable release with not that expensive pricing and not to mention the support that we can get from AWS (previously MWAA had an old Airflow release with no Cloudformation support). But I was wary of being on the bleeding edge of the technology that has not been battle-tested. On the other hand, AWS had a release couple of months ago regarding Airflow on EKS and we all know the scalability of K8s so I thought to give it a try. I will be on the bleeding edge of technology, but at least it will not be a black box.
Assuming we all know parts that constitute Kubernetes and what are helm charts, I will divide the post into two sub-topics i.e.:
Deploying Airflow 2.0 on EKS vs MWAA &Adding DAGs, Variables, Plugins, Connections, etc
Deploying Airflow 2.0 on EKS vs MWAA &
Adding DAGs, Variables, Plugins, Connections, etc
Briefly, the node groups in K8s can be created using AWS Fargate or Node-managed EC2. Both will scale out but the major difference is, with Fargate AWS will manage the worker node scaling and its AMI configuration whereas in Node-Managed we have to do both things but the benefit of it is considerable. Since Fargate does not support daemonsets, with the managed Node group, we don't have to put the agents of monitoring services like Dynatrace, Splunk, or Datadog in a sidecar container and hence more reliability. Additionally, Fargate does not cache container images and hence startup time will be more. Based on my experience, Fargate is more suited for Airflow based workflows where most of it are batches whereas node-managed is more suited for Streaming-related patterns.
A Linux-based environment is set up where your role has desired access. I am using the Ubuntu windows app.
Before creating the cluster we need to install kubectl and eksctl. eksctl will create and manage the cluster whereas kubectl will communicate with the cluster API server in retrieving nodes, pods, namespaces whatever is needed. Thankfully, the binaries are available for both of them.
curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.20.4/2021-04-12/bin/linux/amd64/kubectlchmod +x ./kubectlmkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$PATH:$HOME/bin
And for eksctl,
curl — silent — location “https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz” | tar xz -C /tmpsudo mv /tmp/eksctl /usr/local/bin
Now let's create a k8 cluster using Fargate with the following command. It will take about 15–20minutes and one can check in the cloudformation about the resources (Vpc, subnets, etc) it will be creating.
eksctl create cluster \ — name test-cluster \ — region ap-southeast-2 \ — fargate
And it will show something like:
... [✓] EKS cluster “test-cluster” in “ap-southeast-2" region is ready
Next, let's create a node-managed cluster, and here comes the difference. We need to first create the pub private key pair in the local to be used in EC2 connection.
aws ec2 create-key-pair — region us-west-2 — key-name myKeyPaireksctl create cluster \ — name eksctl-airflow-cluster \ — region ap-southeast-2 \ — with-oidc \ — ssh-access \ — ssh-public-key <your-key> \ — managed
One can check in the CFN about node groups created.
By running the below we can confirm the installation and it should show the nodes present in the cluster:
kubectl get nodes -o wide
Next, we need to update the kubeconfig so that whenever we run kubectl, it should point out to the right cluster.
aws eks — region ap-southeast-2 update-kubeconfig — name eksctl-airflow-cluster
Next, is to create the namespace so that we can deploy the airflow in it. It is just an abstraction to maintain the related resources in one place much like a stack. One cluster can have many namespaces that can communicate with each other.
kubectl create namespace airflow
Before creating airflow in the namespace, we need to install helm
$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3$ chmod 700 get_helm.sh$ ./get_helm.sh
and the following will create airflow resources and you can view it at localhost:8080 once it gets completed. We can give any release_name as you want.
helm repo add airflow-stable https://airflow-helm.github.io/chartshelm repo updatehelm install RELEASE_NAME airflow-stable/airflow  — namespace airflow \ — version 8.3.0
You can find other chart versions at https://github.com/airflow-helm/charts/releases corresponding to the latest Airflow releases.
Before checking the GUI, we need to forward the 8080 port from the pod to the localhost using:
export POD_NAME=$(kubectl get pods — namespace NAMESPACE -l “component=web,app=airflow” -o jsonpath=”{.items[0].metadata.name}”)kubectl port-forward — namespace NAMESPACE $POD_NAME 8080:8080
Now here comes the interesting part, helm charts give values.yml where all the airflow configurations, variables, dagbag folders, etc are mentioned which help in updating the actual airflow.cfg. And hence the only thing left now is to give it the desired values and update using the below command:
helm upgrade airflow airflow-stable/airflow — namespace airflow — version 8.3.0 — values values.yml — debug
The following is a part of values.yml where we can specify the python modules that need to be installed
airflow:    extraPipPackages:         — “pandas”        — “awscli”
Similarly, the following is a part of values.yml where persistent volume and claim are specified that we got after deploying efs.
airflow:    extraVolumeMounts:        — name: airflow-efs-dag          mountPath: /opt/efs/     extraVolumes:        — name: airflow-efs-dag          persistentVolumeClaim:              claimName: efs-storage-claim
And the following will be its dagbag:
dags:    path: /opt/efs/dags
Interesting to know is that MWAA has a similar setup where it asks us to mention all these details in the Cloudformation template itself. To add an airflow variable, MWAA uses the environment variables, and to update env we need to mention it in the CFN. Since AWS SSM is the backend of MWAA, we can specify the key name of the connection containing the conn_string, and MWAA will pick it using the key name and deploy it in Airflow. And as far as the DAG, Plugins folders, and requirements.txt are concerned, we need to give it an s3 folder but the problem is all should be in the same bucket. Not a bottleneck for anyone who is starting. MWAA uses RDS or Aurora at the backend but it does not let the user fiddle around it. It only uses Celery Executor and provides three types of cluster based on different machine types. In all, it can only support 1000 dags running. Hence, MWAA is good for SMEs but not for large-scale companies. At the time of writing, one of the drawbacks of using MWAA is, even if you update an airflow variable through CFN, the scheduler will be stopped at the time of deployment and takes about 10–20min to get it running. And hence a proper release strategy needs to be thought of where no or fewer dags are running whenever there is a new build to be deployed. EKS also takes a couple of minutes in completing helm upgrade but that is more transparent than MWAA as it gives logs with it.
Also, in the case of EKS, we need to integrate EFS and RDS by ourselves which should not be much of a problem, and then any CICD can push the updated DAGs from the code repository to EFS.
values.yml of airflow helm: https://github.com/airflow-helm/charts/blob/main/charts/airflow/values.yaml
While using k8 executor in Airflow, each of the tasks gets to run on the pod whereas the airflow can be running on a separate machine. However in EKS, all the parts of airflow i.e. airflow db, scheduler, etc are running in separate pods and hence more HA and reliability are there.
If you have made it through till now then congratulations you have learned how to deploy Airflow 2.0 on EKS using fargate and node-managed ec2. We have also learned how to customize values.yml that will have airflow config and other provisions like specifying variables, connections, etc. In my opinion, everything will be kubernetes in a matter of time where we have the advantage of separating storage from computing with potentially non-finite amount of computing available on tap. It all depends on what part of the journey you are at in understanding & implementing Data Engineering Landscape.