Set up an Airflow Environment on AWS in Minutes
By: Brent Lemieux
Tags: AWS, Data Engineering, Data Science, Airflow, Machine Learning

Set up an Airflow Environment on AWS in Minutes

Apache Airflow is a powerful platform for scheduling and monitoring data pipelines, machine learning workflows, and DevOps deployments. In this post, we’ll cover how to set up an Airflow environment on AWS and start scheduling workflows in the cloud.

Photo by Ashkan Forouzani on Unsplash

This is part one of a series, Scheduling Machine Learning Workflows. Airflow allows you to schedule your data pulls and featuring engineering pipelines, model training, model deployment, and much more. The content in this tutorial is applicable to anyone who wants to learn how to create, schedule, and monitor their production workflows in an Amazon Airflow environment — not just machine learning engineers.

Set up an Airflow Environment on AWS (this post) More coming soon…

Motivation

Whether you’re an aspiring to a career in engineering or data science, the skill-set of programmatically processing data and building systems to consume that data downstream (e.g. machine learning models) will likely be vital to your success. There’s a reason why these skills are in hot demand.

Why Airflow?

Airflow is one of the most commonly used tools for orchestrating ETL jobs, machine learning pipelines, and DevOps deployments.

When I first started out in machine learning, my (un)management of workflows makes me cringe today. I had jobs scheduled with cron in my own environment on a large, on-premise computer. There were no alerts for when my jobs failed and no UI to monitor progress — I had to manually check the logs each day! When I was on vacation, my co-workers were subjected to this brutal process.

Airflow makes your workflows much more transparent. You can track your jobs progress in the UI and easily configure alerts for when something fails. Teams can easily share an Airflow environment, so you’re not always required to be on-call for your productions jobs.

Why Amazon Managed Airflow?

Up until recently, creating and maintaining an Airflow environment was fairly complex, even for experienced engineers. The Airflow platform includes a front-end webserver, a scheduler service, executors, and a backend database — all of which must be configured. On top of this, Airflow must be connected with other services, like Amazon EMR and S3, in order to utilize them in your pipelines.

When I first started using Airflow, I naively tried to standup my own implementation (this was a disaster). I quickly realized that managed Airflow was the way to go.

Before Amazon launched Managed Workflows with Apache Airflow, there were some other options in the space. However, with Amazon, you have the benefit of seamlessly integrating your workflow management solution with your other resources on AWS. Plus, AWS isn’t going away anytime soon, so you don’t have to worry about the stability of your pipeline management system hinging on another company staying in business. If your interested in learning how to setup an Airflow environment to help you get a job, to me, AWS is the place to start. More companies use AWS than any other cloud provider.

Amazon helps you keep your Airflow version up to date. Minor version updates and patches are handled automatically and allows you to schedule major updates.

Let’s Get Started

WARNING: Creating a managed Airflow environment will cost money. If you’re just doing this for your own education, I recommend setting up the environment, loading and running your DAGs, then deleting your resources. See more on pricing here.

Steps

AWS account setup.

Create a test DAG and upload it to S3.

Write a requirements.txt file to include open source packages in your environment.

Create an Airflow environment in the AWS console.

Access the Airflow UI.

AWS Account Setup

First things first, you’ll need an AWS account if you don’t already have one. Complete steps one and four in this tutorial to setup your account with an IAM admin user and an S3 bucket.

Create a Test DAG

Let’s create a simple Airflow DAG to test in our soon to be created environment.

# test_dag.py # This code borrows heavily from https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html from datetime import timedelta

import logging from airflow import DAG

from airflow.operators.python_operator import PythonOperator

from airflow.utils.dates import days_ago

'owner': 'airflow',

'depends_on_past': False,

'email': ['

'email_on_failure': True,

'email_on_retry': False,

'retries': 1,

'retry_delay': timedelta(minutes=5),

} default_args = {'owner': 'airflow','depends_on_past': False,'email': [' brent@brentlemieux.com '],'email_on_failure': True,'email_on_retry': False,'retries': 1,'retry_delay': timedelta(minutes=5), dag = DAG(

'simple_demo',

default_args=default_args,

description='A simple DAG with a few Python tasks.',

schedule_interval=timedelta(days=1),

start_date=days_ago(2),

tags=['example'],

) ### PYTHON FUNCTIONS

def log_context(**kwargs):

for key, value in kwargs.items():

logging.info(f"Context key {key} = {value}") def compute_product(a=None, b=None):

logging.info(f"Inputs: a={a}, b={b}")

if a == None or b == None:

return None

return a * b ### OPERATORS

t1 = PythonOperator(

task_id="task1",

python_callable=log_context,

dag=dag

) t2 = PythonOperator(

task_id="task2",

python_callable=compute_product,

op_kwargs={'a': 3, 'b': 5},

dag=dag

) t1 >> t2

We need to upload this DAG to our S3 bucket. This is straightforward to do in the AWS console — simply search for S3 in the console, navigate to the folder you’d like to upload to, and click Upload . Alternatively, you can use the AWS CLI to upload your DAG.

Let’s create this path in our bucket and and upload our test DAG there: s3://<your-bucket>/airflow/dags/<your-dag.py> .

Configure Open-Source Packages for the Airflow Environment

Next, we’ll make a requirements.txt file where we’ll specify the open source packages we’d like installed in our environment.

# requirements.txt example

apache-airflow[amazon]==1.10.12

boto3==1.17.44

Upload your requirements file to your S3 bucket in the following location: s3://<your-bucket>/airflow/requirements.txt .

Create the Airflow Enviroment

Now we’re ready to create our environment!

Navigate to Managed Apache Airflow in the AWS console and click Create environment .

2. Name your environment and select your Airflow version (I recommend you choose the latest version).

3. Add your S3 bucket, your DAGs path, and requirements.txt path, then click Next .

4. Set the Web server access to Public network and check Create new security group . Next, click Create MWAA VPC .

5. In the CloudFormation template, leave the defaults and click Create stack .

6. You’ll need to wait a few minutes for your VPC to be created, then select your new VPC.

7. Select your environment class. I recommend selecting mw1.small and scaling up from there if needed.

8. Configure your Monitoring, Encryption, and Airflow configuration options— if you don’t know what to choose just leave the defaults for now.

9. Unless you know what permissions you need, choose the option to Create a new role and AWS will create a role with the necessary permissions for you.

10. Click Next , review your selections, and click Create environment ! It will take 20–30 minutes to spin up your environment.

Access the Airflow UI