How to deploy Airflow on AWS: best practices | by André Sionek | Towards Data Science

I created a repo to deploy Airflow on AWS following software engineering best practices. You can go straight there if you don’t feel like reading this post. But I do describe some things you might find useful here.
Run a docker-compose command and voíla, you have Airflow running on your local environment, and you are ready to develop some DAGs. After some time you have your DAGs (and Airflow) prepared for deployment on a production environment. Then you start searching for instructions on how to deploy Airflow on AWS. Here’s what you’ll probably find:
No instructions on Airflow documentation.
Some posts, like this one, teach you how to deploy on AWS ECS. Quite an interesting approach. The problem is that the whole tutorial is based on creating resources by point-and-click on the AWS console. Trust me; you don’t want to go that route for deploying in production. Just imagine the nightmare for creating three different environments (dev, staging and production) and having to repeat the process three times. Now imagine updating the environments and keeping them in sync. Picture how you could easily spend a whole week fixing a bug caused by a resource that was deleted by mistake.
This other post is relatively recent and has a superior approach, but it still creates resources (like the ECS cluster) using AWS CLI. A little better than console point-and-click, but still not production-ready.
Other articles, might mention using Infrastructure as Code, which would solve the problems mentioned above. However, they are very shallow in technical details and implementation. So, despite offering a good overview and best practices, they are not practical for someone without DevOps experience.
This repo on GitHub is probably the closest you’ll get from a proper implementation of Airflow on AWS following software engineering best practices. But it still lacks some basic stuff like autoscaling of webservers and workers or a way to configure settings such as RDS instance type without having to dig through Terraform code.
You might have a look on page 2 of your Google search, but if a good result doesn’t show up on the first page, then it probably means that it doesn’t exist.
Just clone the repo, follow the few instructions and install the requirements described on the README file and then run:
make airflow-deploy
This will deploy an Airflow instance to your AWS account!
This repo implements all infrastructure using AWS Cloudformation. Inside the /cloudformation directory you’ll find all templates to create the infrastructure needed to run Airflow. The good thing is you don’t need to worry about learning Cloudformation to do a simple deploy because in the root there is a service.yml to help you.
Let’s say you want to whitelist specific IPs (such as your office IP) to have access to Airflow UI. The only thing you need to do is change a few lines in service.yml:
If you want to change the Airflow database instance type, you also go to service.yml:
You want to just the Airflow workers CPU and memory? Fine-tune autoscaling? service.yml is the one-stop-shop.
Don’t hardcode your passwords is Software Engineering 101. But sometimes it is hard to automate deployments and create passwords at runtime without hardcoding then somewhere. In this Airflow repo, we use AWS Secrets Manager to help us solve that.
Airflow Metadata DB
Our Postgres database that will hold Airflow metadata is one of the resources that require an admin username and password. Using AWS Secrets Manager, a strong random password is created at deploy time and attached to the cluster. To get the password value, you have to log in to your AWS account and go to Secrets Manager.
It is also possible to implement automatic password rotation with Secrets Manager, but it was not implemented for this project.
Fernet Key
Airflow uses a Fernet Key to encrypt passwords (such as connection credentials) saved to the Metadata DB. This deployment generates a random Fernet Key at deployment time and adds it to Secrets Manager. It is then referenced in the Airflow containers as an environment variable.
One of the biggest challenges of putting Airflow in production is dealing with resources management. How to avoid crashing the webserver if there is a usage peak? Or what to do if a particular daily job requires more CPU/memory?
Autoscaling solves those issues for you. In this repository, you can easily configure thresholds and rest assured that your infrastructure will scale up and down to meet demand.
In a production setup, you will want to deploy your code to different environments. Let’s say you’ll need: prod, stage and dev.
This repo allows you to deploy the same code to different environments by just changing one environment variable, that could be automatically inferred on you CI/CD pipeline. To change the environment, do:
export ENVIRONMENT=dev; # this will deploy airflow to dev environmentmake airflow-deploy;
The beauty of Airflow is the ability to write workflows as code. It means that you will change DAGs code much more often than you change infrastructure. With this deployment of Airflow, you will submit changes to your DAGs, and it won’t try to redeploy the infrastructure for you.
The only thing you want to do is build a new Airflow image, push it to ECR and then update your ECS service to load the latest image. To achieve that, just run:
make airflow-push-image;
It is not the case here, but you could even have your DAGs sitting on a separate repository. It would separate infrastructure from software even more.
Tagging resources will allow us to create automated alerts, identify ownership and track infrastructure costs easily. That’s why this Airflow repository tags all resources.
You should add the deployment process to your CI/CD pipeline. To run some automated tests, I’m using GitHub Actions (but your company might be using other tools such as CircleCI or Jenkins).
You can follow a similar process to automate your deploy and tests. Have a look at the tests workflow to get some inspiration.
Running Airflow on AWS with the default configurations (also considering cluster is not scaling up) should cost from 5 to 7 US dollars per day. Depending on the region you are deploying.
This cost can be further reduced by lowering CPU and memory on service.yml and also changing the minimum number of workers. The default settings would allow Airflow to run quite a few dags before needing to increase resources. Fine-tune and find what works best for your use case.
I hope this post (and the repository) helps you to easily productionize Airflow. If you have any questions, suggestions or requests, please reach out to me on LinkedIn, or open an issue on the repo. Also, you are welcome to open PRs and collaborate with the project!