Migrating Airflow from Amazon ECS to Astronomer
By: Johnson K
Tags: Apache Airflow, Astronomer, Elastic Container Service, Mage, Airflow

Companies of any size working with large amounts of data use Airflow to run jobs with complex dependencies on a regular schedule. Whether creating a new Airflow project or working off an existing one, using an Airflow managed service can make it easier to get Airflow up and running at scale. Here at Mage, we use Astronomer Cloud.

Co-author: JP Hinek

Topics Discussed:

What is Airflow? Our previous setup Challenges running Airflow in ECS Managed Airflow alternatives Our current setup Migrating to Astronomer Cloud Deploying Airflow to production using Astronomer Cloud

With Airflow, be the captain, conductor, and manager of tasks that run themselves.

What is Airflow?

Apache Airflow is an open-source data orchestration tool for programmatically creating workflows using Python. A workflow includes directed acyclic graphs (DAGs) which have tasks connected to one another but never in a closed loop.

Why should you care?

Workflows can automate and streamline repeatable tasks, increasing productivity and efficiency for your project or business. Creating these workflows without a tool like Airflow can be a difficult and increasingly unmanageable endeavor as the number of tasks grows and the dependencies between tasks become more and more complicated.

If you need to process large amounts of data and have jobs with many steps running on a recurring schedule, Airflow can make your life a lot easier.

With Airflow, let repeatable operations run by themselves.

How we use Airflow at Mage

Airflow is an essential part of our software stack. Mage is a tool that helps developers use machine learning and their data to make predictions. We use Airflow to orchestrate the steps required in preparing training data, building new machine learning models, deploying machine learning models, processing streamed data, and more.

Airflow is an important part of making sure steps in our workflow happen in a certain order and on a recurring basis.

Options for running Airflow in production

There are many different methods for running Airflow in production, depending on the needs of your project. Though Airflow comes with SQLite by default, it will need a production database like PostgreSQL or MySQL for storing metadata. Amazon’s Relational Database Service (RDS) or Google’s Cloud SQL are two options for hosting the database.

The other components of Airflow can be hosted in different cloud providers or on your own bare-metal server. Here is a non-exhaustive list of various cloud services you can use (get ready for lots of acronyms):

Amazon Web Services

Elastic Compute Cloud (EC2)

Elastic Container Service (ECS)

Elastic Kubernetes Services (EKS)

Google Cloud Platform:

Google Compute Engine (GCE)

Google Cloud Run

Google Kubernetes Engine (GKE)

Microsoft Azure

Azure Virtual Machines (VMs)

Azure Container Instances (ACI)

Azure Kubernetes Service (AKS)

Heroku

Our previous setup — Amazon ECS (Elastic Container Service)

For our previous setup, we used an Amazon ECS cluster running Airflow version 1.10.14. In this ECS cluster, we utilized five services: Workers, Webserver, Scheduler, Flower, and the Celery Executor.

Simplified Airflow architecture

In addition, we had a Redis cluster in Amazon ElastiCache and a PostgreSQL database hosted in Amazon RDS. Refer to the Appendix at the bottom of this post for more details on the Airflow components.

Challenges running Airflow in ECS

As the number of DAGs in our Airflow cluster grew, we noticed that DAGs would sometimes have issues running successfully, or there would be problems with the scheduler. We also had unnecessary costs due to over-provisioned Airflow workers, and it became a hassle to find the right resource allocation through the Amazon ECS UI.

Managing the number of Airflow workers and resources allocated to the various AWS services required for Airflow was a bit cumbersome. We needed a way to easily manage these resources and quickly deploy changes to our infrastructure if necessary. We wanted to be able to scale up Airflow task processing efficiently, effectively, and with as little overhead and maintenance from the team as possible, and we were willing to pay for a managed service to help us with that.

Managed Airflow alternatives

Airflow config can be a complicated process; use these tools to help you manage it.

Amazon Managed Workflows for Apache Airflow (MWAA)

MWAA is a managed service in AWS for Apache Airflow. It’s backed by Amazon and meant to integrate with other Amazon services. Google Cloud has their version of a managed service for Airflow in Cloud Composer, but since Mage’s cloud infrastructure is built in AWS (future article on this coming soon), we won’t be mentioning the advantages or disadvantages of using Google Cloud Composer as it wasn’t an option for us.

Pricing for MWAA or Astronomer Cloud isn’t mentioned below because the difference in pricing between the two wasn’t so drastically different that it was a deciding factor in choosing one over the other.

Pros

Autoscaling workers: MWAA is a fully-managed version of Airflow, so it makes it easier to scale up Airflow. If more worker instances are needed to process tasks, MWAA takes care of it automatically and removes workers when they are no longer needed.

Compatibility with other AWS services: MWAA is part the AWS ecosystem and connects well with the tools and other resources required for your workflow if you are already hosting your infrastructure on AWS. For example, it has logging already integrated with Amazon CloudWatch.

Cons

Outdated Airflow Version: At the time of writing (April 2021), the latest Airflow version available in MWAA was 1.10.12. The current latest version of Airflow is 2.0.2. If you wanted to use Airflow 2.0, that’s automatically a dealbreaker for integrating MWAA as your tool for managing Airflow. Though MWAA will inevitably support Airflow 2.0, this may or may not mean that MWAA is slower to adopt the latest Airflow versions.

Required VPC infrastructure: If you require or already have a specific Virtual Private Cloud (VPC) infrastructure in AWS, creating a new MWAA environment may require adjustments to your current VPC or an entirely new VPC. This may not be an issue for you, but it can be a hassle if you just wanted to add the MWAA environment into an existing VPC but weren’t able to due to certain restrictions.

DAGs have to be uploaded to S3: Amazon Simple Storage Service (S3) is used to read all DAGs in MWAA. This could be fine for some workflows, but it could also result in making certain issues harder to debug when new DAGs are deployed. If you want flexibility in where you manage your DAGs and don’t want to go through the added step of uploading them to S3, this requirement for using MWAA may not be worth the additional effort, especially for existing Airflow projects.

Astronomer

Astronomer is a commercial managed Airflow platform that is cloud-agnostic. It can be used with your own cloud environment, whether it is AWS, GCP, or another cloud provider.

“Airflow look at me, I’m the scheduler now.”

Pros

DAGs maintained in Git repositories: You can maintain the folder structure of your current Git repo and dynamically build DAGs without issues potentially caused by importing DAG files to an Amazon S3 bucket. Astronomer allows you to deploy DAGs directly from Git repos.

Cloud-agnostic: Astronomer is not confined to one cloud provider. You can also deploy your Airflow project to Astronomer’s own cloud infrastructure.

Easy initial setup: Astronomer Cloud setup and configuration is fairly straightforward following their documentation and doesn’t require additional security configuration regarding access keys or VPCs, since the Airflow project is hosted in Astronomer’s VPC. However, you can configure your own security settings if you use Astronomer Enterprise.

Customer support: Astronomer users have access to a customer support portal in which they can get quick responses to issues (typically within a business day or two or faster if needed, though it will cost extra) and help with troubleshooting Airflow issues. Astronomer’s business is built around Airflow so their customer support is specialized in Airflow.

CI/CD integration: Airflow deployments through Astronomer can be integrated into CI/CD pipelines using several different CI/CD tools like CircleCI, Jenkins, etc.

Quick deployments: Astronomer’s UI allows you to adjust resources allocated to the scheduler, webserver, and workers all on one page and deploy the changes within a few minutes. Not having to go through several screens to make these changes is convenient if you’re using Astronomer Cloud.

Cons

Limited logs visibility: There may be issues viewing the worker logs, as well as historical scheduler or webserver logs, through the Astronomer UI in Astronomer Cloud.

Log retention period: The log retention period for Astronomer Cloud is 15 days, so if you want access to logs older than that, you may not be able to view them through the Astronomer UI. If you have remote logging enabled, you can still have access to task logs older than 15 days.

Paying for idle resources with the Celery executor: If you’re using the Celery executor and have a lot of resources allocated to your workers but only use your workers for a brief period during the day, you still have to pay for the full capacity of the workers even when they’re not being fully utilized. You do have the option to use the Kubernetes executor, though.

Astronomer currently has two offerings: Cloud and Enterprise.

Astronomer Cloud:

Astronomer Cloud manages all of the infrastructure required to run Airflow. Your Airflow project is run on a Kubernetes cluster and hosted in Astronomer’s Google Cloud environment.

Astronomer Enterprise:

Astronomer Enterprise allows you to run and manage Airflow in your own cloud environment while still utilizing features and support from Astronomer. The enterprise version can be installed on the following Kubernetes environments: AWS Elastic Kubernetes Services (EKS), GCP Google Kubernetes Engine (GKE), or Microsoft Azure Kubernetes Service (AKS).

Astronomer Enterprise gives you more control over your infrastructure hosting Airflow, so you can customize your security settings, such as IAM roles, authentication, and VPC configuration. In addition, you have access to Grafana (a monitoring tool for Airflow metrics), Kibana (a visualization tool for your Airflow logs), and other tools.

Our current setup — Astronomer Cloud

Astronomer is one tool we use to bring our customers magical experiences

We ultimately ended up going with Astronomer Cloud as our managed service for hosting our Airflow cluster. We did try setting up our Airflow cluster in Amazon MWAA initially because all of our infrastructure was already using AWS, so we thought the transition would be easier.

However, we quickly ran into issues configuring the MWAA environment because we had to create a separate VPC from what our other AWS services were hosted in. Our Airflow setup at the time required a connection to our Redis cluster in Amazon ElastiCache, but we could not make the connection because the Redis cluster was in a different VPC, so we likely needed to create a VPC peering connection. We also had issues building all of our DAGs because we have many dynamically-generated DAGs, and the MWAA workflow requiring DAG files to be uploaded to S3 would have required a lot of big changes to our code.

As for Astronomer Cloud versus Astronomer Enterprise, Astronomer Cloud made more sense for us because we wanted a solution that managed most, if not all, of the infrastructure required for Airflow for us. We had to make some adjustments as mentioned later in this post, but we wouldn’t consider them major changes. As our company grows, Astronomer Enterprise is still an option, and it’s common for companies to start off using Astronomer Cloud and then later switch to Astronomer Enterprise as their needs change.