Set up an Airflow Environment on AWS in Minutes | by Brent Lemieux | Towards Data Science

Apache Airflow is a powerful platform for scheduling and monitoring data pipelines, machine learning workflows, and DevOps deployments. In this post, we’ll cover how to set up an Airflow environment on AWS and start scheduling workflows in the cloud.
This is part one of a series, Scheduling Machine Learning Workflows. Airflow allows you to schedule your data pulls and featuring engineering pipelines, model training, model deployment, and much more. The content in this tutorial is applicable to anyone who wants to learn how to create, schedule, and monitor their production workflows in an Amazon Airflow environment — not just machine learning engineers.
Set up an Airflow Environment on AWS (this post)More coming soon...
Set up an Airflow Environment on AWS (this post)
More coming soon...
Whether you’re an aspiring to a career in engineering or data science, the skill-set of programmatically processing data and building systems to consume that data downstream (e.g. machine learning models) will likely be vital to your success. There’s a reason why these skills are in hot demand.
Airflow is one of the most commonly used tools for orchestrating ETL jobs, machine learning pipelines, and DevOps deployments.
When I first started out in machine learning, my (un)management of workflows makes me cringe today. I had jobs scheduled with cron in my own environment on a large, on-premise computer. There were no alerts for when my jobs failed and no UI to monitor progress — I had to manually check the logs each day! When I was on vacation, my co-workers were subjected to this brutal process.
Airflow makes your workflows much more transparent. You can track your jobs progress in the UI and easily configure alerts for when something fails. Teams can easily share an Airflow environment, so you’re not always required to be on-call for your productions jobs.
Up until recently, creating and maintaining an Airflow environment was fairly complex, even for experienced engineers. The Airflow platform includes a front-end webserver, a scheduler service, executors, and a backend database — all of which must be configured. On top of this, Airflow must be connected with other services, like Amazon EMR and S3, in order to utilize them in your pipelines.
When I first started using Airflow, I naively tried to standup my own implementation (this was a disaster). I quickly realized that managed Airflow was the way to go.
Before Amazon launched Managed Workflows with Apache Airflow, there were some other options in the space. However, with Amazon, you have the benefit of seamlessly integrating your workflow management solution with your other resources on AWS. Plus, AWS isn’t going away anytime soon, so you don’t have to worry about the stability of your pipeline management system hinging on another company staying in business. If your interested in learning how to setup an Airflow environment to help you get a job, to me, AWS is the place to start. More companies use AWS than any other cloud provider.
Amazon helps you keep your Airflow version up to date. Minor version updates and patches are handled automatically and allows you to schedule major updates.
WARNING: Creating a managed Airflow environment will cost money. If you’re just doing this for your own education, I recommend setting up the environment, loading and running your DAGs, then deleting your resources. See more on pricing here.
AWS account setup.
Create a test DAG and upload it to S3.
Write a requirements.txt file to include open source packages in your environment.
Create an Airflow environment in the AWS console.
Access the Airflow UI.
First things first, you’ll need an AWS account if you don’t already have one. Complete steps one and four in this tutorial to setup your account with an IAM admin user and an S3 bucket.
Let’s create a simple Airflow DAG to test in our soon to be created environment.
# test_dag.py# This code borrows heavily from https://airflow.apache.org/docs/apache-airflow/stable/tutorial.htmlfrom datetime import timedeltaimport loggingfrom airflow import DAGfrom airflow.operators.python_operator import PythonOperatorfrom airflow.utils.dates import days_agodefault_args = {    'owner': 'airflow',    'depends_on_past': False,    'email': ['brent@brentlemieux.com'],    'email_on_failure': True,    'email_on_retry': False,    'retries': 1,    'retry_delay': timedelta(minutes=5),}dag = DAG(    'simple_demo',    default_args=default_args,    description='A simple DAG with a few Python tasks.',    schedule_interval=timedelta(days=1),    start_date=days_ago(2),    tags=['example'],)### PYTHON FUNCTIONSdef log_context(**kwargs):    for key, value in kwargs.items():        logging.info(f"Context key {key} = {value}")def compute_product(a=None, b=None):    logging.info(f"Inputs: a={a}, b={b}")    if a == None or b == None:        return None    return a * b### OPERATORSt1 = PythonOperator(    task_id="task1",    python_callable=log_context,    dag=dag)t2 = PythonOperator(    task_id="task2",    python_callable=compute_product,    op_kwargs={'a': 3, 'b': 5},    dag=dag)t1 >> t2
We need to upload this DAG to our S3 bucket. This is straightforward to do in the AWS console — simply search for S3 in the console, navigate to the folder you’d like to upload to, and click Upload. Alternatively, you can use the AWS CLI to upload your DAG.
Let’s create this path in our bucket and and upload our test DAG there: s3://<your-bucket>/airflow/dags/<your-dag.py>.
Next, we’ll make a requirements.txt file where we’ll specify the open source packages we’d like installed in our environment.
# requirements.txt exampleapache-airflow[amazon]==1.10.12boto3==1.17.44
Upload your requirements file to your S3 bucket in the following location: s3://<your-bucket>/airflow/requirements.txt.
Now we’re ready to create our environment!
Navigate to Managed Apache Airflow in the AWS console and click Create environment.
Navigate to Managed Apache Airflow in the AWS console and click Create environment.
2. Name your environment and select your Airflow version (I recommend you choose the latest version).
3. Add your S3 bucket, your DAGs path, and requirements.txt path, then click Next.
4. Set the Web server access to Public network and check Create new security group. Next, click Create MWAA VPC.
5. In the CloudFormation template, leave the defaults and click Create stack.
6. You’ll need to wait a few minutes for your VPC to be created, then select your new VPC.
7. Select your environment class. I recommend selecting mw1.small and scaling up from there if needed.
8. Configure your Monitoring, Encryption, and Airflow configuration options— if you don’t know what to choose just leave the defaults for now.
9. Unless you know what permissions you need, choose the option to Create a new role and AWS will create a role with the necessary permissions for you.
10. Click Next, review your selections, and click Create environment! It will take 20–30 minutes to spin up your environment.
Once your environment is ready, simply click Open Airflow UI and watch you’re up and running!
Now that you know how to set up an Airflow cluster, try your hand at creating more Airflow DAGs and testing them in your environment. In the near future I’ll be creating more Airflow tutorials, covering topics like running Spark jobs in Amazon EMR.
Thank you for reading! Please let me know if you liked the article or if you have any critiques. If you found this guide useful, be sure to follow me, so you don’t miss my future articles.
If you need help with a data project or want to say hi, connect with me on LinkedIn. Cheers!