Managed Apache Airflow on AWS — New AWS Service For Data Pipelines | by Anna Geller | Towards Data Science

Apache Airflow has been initially released as an open-source product in 2015 [1]. Since its creation, it gained a lot of traction in the data engineering community due to its capability to develop data pipelines with Python, its extensibility, a wide range of operators, and an open-source community. Despite its popularity, deploying Airflow to a robust and secure production environment has always been challenging.
In fact, there are companies (such as Astronomer), consultants (a.o. Polidea and GoDataDriven), and cloud services (such as Google Cloud Composer or many AWS Marketplace offerings) that specialized in offering enterprise support for deploying and managing Airflow environments. AWS now enters this market, too.
The new fully managed service from AWS lets you create a production-ready Airflow environment within a few clicks in the management console. In this article, we’ll look at how it works and investigate how it differs from competing managed Airflow offerings.
The main benefit and selling point of MWAA is convenience: a managed service with elastic worker-node capacity that allows you to deploy your DAGs without having to worry about the underlying infrastructure. This means that you no longer need to monitor and manually scale your Celery workers to meet your workflows' demand. So far, having elastic worker nodes with Airflow was only possible using KubernetesExecutor and KEDA (Kubernetes Event-Driven Autoscaler).
With MWAA, you don’t choose Airflow executor — MWAA only supports CeleryExecutor with an autoscaling mechanism implemented under the hood. It’s no surprise that AWS-native SQS is leveraged as a message queue to process the scheduled Celery worker tasks.
Airflow relies on a metadata database that stores information about your workflows. AWS utilizes RDS Aurora (Postgres) for that purpose.
MWAA uses S3 as a storage layer for the code, i.e., DAG files, plugins code, and requirements.txt to install additional Python packages within the Airflow environment.
Besides the autoscaling of worker node capacity, one of the most considerable advantages of MWAA is the fact that it’s a managed service. You don’t need to monitor webserver, worker nodes, and scheduler logs to ensure that all components within your environment are working — AWS is responsible for keeping your environment up and running at all times. The same is true for security patches and upgrades to new Airflow versions.
The integration with other AWS services makes it easier to manage communication between Airflow and other services running within your VPC. For instance, instead of maintaining and manually rotating credentials, you can now leverage IAM roles for more robust management of permissions.
Another useful feature is a centralized storage of configuration options. Airflow is configured to a large extent by setting variables in the airflow.cfg file. With MWAA, you can manage those settings directly from the management console. Normally, after having changed your configuration variables, you would have to restart your scheduler and worker nodes to apply the changes. With MWAA, it’s performed under the hood without bringing down your Airflow environment.
The functionality of MWAA environments can be extended by using plugins — you simply need to upload plugins.zip to your S3 bucket to make custom operators, hooks, and sensors available to all your DAGs. Example structure of plugins.zip [3]:
__init__.py    |-- airflow_plugin.pyhooks/    |-- __init__.py    |-- airflow_hook.pyoperators/    |-- __init__.py    |-- airflow_operator.pysensors/    |-- __init__.py    |-- airflow_sensor.py
One of my favorite aspects of MWAA is how easy it is to configure logging to CloudWatch. Setting this up yourself would require a lot of work to configure a CloudWatch Agent on all instances to stream the logs to CloudWatch, and ensure proper log groups for all components (scheduler logs, worker nodes logs, web server logs, and the actual logs from your tasks).
My second favorite feature is the integration of MWAA with IAM: only authorized IAM users can log into your Airflow UI. This is enabled by default without having to implement any custom Auth logic. When I open the link to the Airflow UI within a private browser window, we are not able to access the UI until we sign in with a user who has access to the AWS management console:
From the security perspective, I would definitely recommend using MWAA rather than deploying Airfow yourself with EC2. AWS ensured that any data processed within MWAA is by default encrypted with KMS, and you get the Single-Sign-On that works out of the box. Similarly, you don’t need to worry about Security Groups, and a domain name is automatically assigned to your environment.
From the DevOps perspective, MWAA is attractive since establishing a CI/CD pipeline for your DAGs is as easy as making sure that any push to the master branch triggers upload of the code to s3://airflow-BUCKET/dags.
Lastly, AWS also promises support for containerized workloads with AWS Fargate and enables easy consolidated pricing for all your Airflow components by using tags assigned to your MWAA, S3, and CloudWatch resources.
The overall process of creating an environment is to configure:
VPC with two private subnets (via CloudFormation template),
S3 bucket that will be used as a storage of your DAG files (it must be a bucket that starts with “airflow-” and with versioning enabled!), and optionally to upload plugins.zip to use custom operators, sensors, and hooks, as well as to upload requirements.txt to make additional Python packages available within this Airflow environment.
In the demo below, I’m using the AWS management console to create the MWAA Airflow environment, including creating a VPC with two private subnets.
Once the environment is available, you can upload a simple test DAG to the S3 bucket: s3://airflow-mwaa-dev.
aws s3 cp ex_dag.py s3://airflow-mwaa-dev/dags/
After uploading to S3, we can find the DAG in the UI:
As with any cloud service, you need to know what such a managed service will cost and how it differs from a self-managed environment. By deploying everything yourself and perfectly matching the compute resources to your needs, you could have some potential savings compared to a managed service. However, from my perspective, the autoscaling feature of your Celery worker nodes will most likely already provide cost savings compared to a self-hosted Airflow environment with idle compute resources.
Also, one needs to consider the time savings that a managed service enables — you no longer need to spend time monitoring, patching, and restarting stuck webserver and scheduler instances, fixing broken Celery queues, and matching the worker nodes capacity to the needs of your workflows.
AWS charges separately for the MWAA instance, worker nodes, and metadata database storage. Also, S3 and CloudWatch charges apply based on your region and usage.
When you delete your Airflow MWAA environment, please make sure to delete the VPC that was configured within the default MWAA configuration via CloudFormation template, especially the NAT Gateway, for which the charges apply hourly. As long as you delete the VPC, make sure that the NAT Gateway is deleted as well to avoid any unnecessary charges. You can see it all from the VPC section in the management console:
Managed services are great in terms of eliminating the headache related to maintaining the environment. However, the disadvantage is that:
you can’t SSH to your scheduler, worker, or web server instances — the underlying servers are abstracted away from you, and you can’t SSH to them. Personally, I see no need to access those as long as everything is working, but some may view it as a constraint. This means that you can access the logs from those instances within CloudWatch, but you cannot, for instance, SSH to the instance and manually install some additional packages.
you can’t access the metadata database directly to query your DAG information. From my perspective, nobody should be accessing this information directly, but rather via an API, so I find this abstraction useful. However, some people may see it as an undesirable constraint. Also, note that lots of information about your DAGs can be accessed from the Browse UI menu and by leveraging the API.
using a specific Airflow version — at the time of writing, MWAA only supports the latest stable version, since AWS saw some security concerns in the previous Airflow versions [5]. If your workflows require a specific Airflow version, MWAA may not work for you.
managing Python and code dependencies — this problem is not really tied to MWAA, but rather to Airflow, overall. If your code needs custom modules and packages, you may prefer to make use of DockerOperator or PythonVirtualEnvOperator. To learn more about those, you can have a look at the following articles:
towardsdatascience.com
medium.com
There are many potential reasons why AWS may have decided to create this managed service.
The first, quite an obvious reason, could be:
So far, Google Cloud Platform has been (subjectively) offering the most comprehensive range of products with respect to data engineering, offering services such as Big Query (serverless, cost-effective, and ridiculously feature-rich DWH), Cloud Composer (Apache Airflow), Cloud Dataflow (Apache Beam), and many more.
AWS could provide corresponding services for storing data such as Redshift (DWH), AWS Glue (metadata catalog), and Amazon Athena (serverless Presto), but the missing part was an orchestrator to glue those data services together. With MWAA, this missing piece can potentially be filled.
Side note: You may say that AWS has Simple Workflow Service or Step Functions. But those services are hard to use to orchestrate data pipelines for ETL and ML, where most people prefer to do it in Python — the language of data.
Apart from that, MWAA makes it easier to build ML pipelines using Sagemaker, as it allows you to combine all preprocessing and model training steps in a repeatable process.
AWS consistently demonstrated its willingness to make open-source technologies more accessible and easier to manage. So far, AWS introduced managed services for:
Kafka (Amazon MSK)
Presto (Amazon Athena)
Hadoop (Amazon EMR)
Open-source relational databases such as Postgres, MySQL, MariaDB (RDS)
Kubernetes (AWS EKS)
Elasticsearch (Amazon Elasticsearch Service)
Kibana (directly integrated within Elasticsearch service)
Apache MXNet, PyTorch, Tensorflow & Jupyter (Sagemaker)
Redis and Memcached (ElastiCache).
Now, AWS is adding to this list Apache Airflow with the MWAA service.
“Customers have told us they really like Apache Airflow because it speeds the development of their data processing and machine learning workflows, but they want it without the burden of scaling, operating, and securing servers,” — Jesse Dougherty, Vice President, Application Integration, AWS. [Source]
AWS is customer-centric and security-aware. According to the above quote from the press release, many customers wanted to have Airflow on AWS, but it was too painful to deploy and maintain. To have a production-ready cloud-based Airflow environment, one needs to consider so many components that many customers likely decided to switch to competitors in order to get an easier and more reliable setup.
Personally, I came across Airflow UIs that were publicly exposed to anyone without RBAC functionality. This happened because some people deployed Airflow via AWS Elastic Beanstalk without proper security mechanisms. This means that anyone in the world could access their environment, code, and data simply by using the admin-UI URL:
http://URL-aws-region.elasticbeanstalk.com/admin/
Thanks to the managed service that introduces security by default (with encryption and Single-Sign-On), hopefully, fewer people will encounter such vulnerabilities in their Airflow deployments.
While Astronomer is specialized in containerized Airflow environments deployed to a Kubernetes cluster, AWS MWAA is leveraging Celery executor and Celery workers deployed to managed EC2 instances running Amazon Linux AMI. Therefore, those two offerings are hard to compare against each other. From a pure feature perspective, Astronomer seems to be offering more due to their astro CLI that makes local development and code deployments easier, and additional features built on top, such as management of different team workspaces and dedicated enterprise support for Airflow.
I doubt that adding a managed Airflow service to the AWS portfolio will put vendors such as Astronomer out of business. Companies still need and often prefer interacting with service providers (or consultants) specialized in one product that they know well. This is why Confluent is still doing great as a company despite the existence of a fully managed service Amazon MSK (Managed Streaming for Apache Kafka). However, providers such as Astronomer may lose significant market share due to the AWS-native managed service that eliminates the need for maintenance of a Kubernetes cluster.
When comparing against Cloud Composer from GCP, the discussion becomes more interesting. While previously, companies, deciding which cloud provider to choose, might have preferred GCP due to their comprehensive data engineering services, they might now opt for AWS. Those services are also very similar in how they work — they both use CeleryExecutor for execution and object storage to store your DAG code, and they both let you easily create Airflow environments with a few clicks from the management console.
The main difference between AWS MWAA and GCP Cloud Composer:
while MWAA automatically scales the number of worker nodes, GCP does not — you need to monitor the worker nodes or Kubernetes pods yourself on GCP to manage the load of your workflows.
Note: this comparison has been created based on publicly available information at the time of writing. Those services are evolving fast — make sure to validate this information with the respective service providers before making any decisions.
In this article, we looked at the AWS fully managed service for Apache Airflow. We discussed its internal architecture, costs, benefits, and possible challenges when using that service. Additionally, we showed how to create your first environment from the management console. We discussed the possible reasons behind why AWS might have decided to incorporate Airflow into their range of supported open-source tools. Finally, we briefly compared MWAA against its competitors.
To answer the question from the subtitle — is AWS MWAA worth trying? It boils down to whether you really want to use Airflow. If you are not sure whether Airflow suits your data engineering needs, have a look at those articles:
towardsdatascience.com
towardsdatascience.com
Thank you for reading.
References:
[1] Wikipedia about Airflow
[2] AWS Blog
[3] AWS MWAA Docs on plugins
[4] AWS Docs
[5] AWS MWAA FAQ