Automating Emails in Apache Airflow: A How-To Guide | by Zach Alexander | Towards Data Science

As a Data Engineer, some of the work I do involves automating data pulls at regular intervals from third-party sources for various teams at the organization. Although I like to think staff are always aware of when a new data pull occurs (ha!), I can’t expect them to remember each time it happens. Because of this, Apache Airflow has been a great service to help send e-mail reminders to interested parties when a data pipeline has completed and a new dataset has been processed.
Airflow not only does a great job orchestrating the data pull itself but works well with Python libraries (such as SMTP) to facilitate the transfer of the email to the proper recipient(s).
In this quick guide below, I’ll walk you through setting up your first directed acyclic graph (DAG) in Apache Airflow that orchestrates this e-mail automation process!
If you don’t know what Apache Airflow is, feel free to first read through these resources below to get a better sense of some of its use cases.
Official Airflow Documentation: https://airflow.apache.org/docs/apache-airflow/stable/index.html
A Good Medium Post on Airflow: https://medium.com/swlh/apache-airflow-in-5-minutes-c005b4b11b26
Mainly, you’ll want to have a basic understanding of tasks, operators, and Airflow’s file structure.
Additionally, e-mail automation in Python with SMTP depends on a properly configured sender e-mail address. Although many different e-mail providers can be used, I’ve found that Gmail makes it very easy.
In order for the sender e-mail to operate correctly with SMTP and Python, you’ll need to follow the instructions here (under the Option 1: Setting Up A Gmail Account for Development). This should be a pretty quick step, where you’ll need to change one of your security settings to “Allow less secure apps to ON”.
When both prerequisites are met, you’ll then want to tackle installing Airflow on your local machine (if you haven’t already).
If/when you feel comfortable with Airflow, you’ll then want to make sure it’s installed locally. If you need some help with this process, here are a few starters for local installations:
Installation on Mac: https://arpitrana.medium.com/install-airflow-on-macos-guide-fc66399b2a9e
Installation via Docker: https://medium.com/@itunpredictable/apache-airflow-on-docker-for-complete-beginners-cf76cf7b2c9a
Installation on Windows 10: https://medium.com/@ryanroline/installing-apache-airflow-on-windows-10-5247aa1249ef
As a quick note, if you are planning to install Airflow on Windows (without Docker), you’ll need to create a Linux subsystem. The instructions should be helpful in the guide above, but it might take a bit of configuration with your local setup to get this working.
Once you have Airflow installed, and you have a good handle on the software itself, we can get started.
If you haven’t already, spin up your webserver by running the following command in a terminal window:
airflow webserver
Additionally, you can spin up the Airflow scheduler in a separate terminal window:
airflow scheduler
You should then be able to navigate to http://localhost:8080 (or whichever endpoint you’ve configured for your local webserver), and find your UI.
To avoid duplication with other posts, and to cut down on the length of the article, I won’t go into detail about uploading dags. Ultimately, when you spin up the webserver you should see your dags load into the UI if your directory and environment are structured correctly. If you are spinning up your webserver for the first time, you may see some sample dags in the UI if you configured this option in your airflow.cfg file.
Now, with the webserver running on your local machine, we can create the DAG that’ll send an e-mail. To help guide you through this process, I’ve posted my DAG file on GitHub:
https://github.com/zachalexander/airflow-medium-post
Feel free to clone the repository and work through this simultaneously, or read through the guide below and then use the code to integrate this into your DAGs later on!
Important Note: For those that are already using Airflow regularly, since the DAG file contains a just a few functions, you can integrate these into any pipeline you already have set up (and add it on as a task). However, for the sake of this walkthrough, I’ll create this as a separate, one-step dag for simplicity.
To create our e-mail automation DAG, we can navigate to the dags folder in your Airflow project, which should be structured similar to my GitHub repo. You can either use my existing “email_automation.py” file or you can create your own blank python file. An example of the directory structure is below:
airflow  dags/    email_automation.py
Inside of this python file, you can add the following imports at the top:
from airflow import DAGfrom airflow.operators.python_operator import PythonOperatorfrom datetime import timedeltaimport smtplib, sslfrom email.mime.text import MIMETextfrom email.mime.multipart import MIMEMultipart
The first import allows for DAG functionality in Airflow, and the second allows for Airflow’s Python Operator, which we’ll use to initiate the e-mail later on. Importing timedelta will help us regulate a timeout interval in the occurrence of our DAG taking too long to run (Airflow best practice). Then, we’ll import some e-mail and SMTP libraries that will be essential to sending the e-mail to our recipients.
After this, we can then add our first function declaration, which is named send_email_basic():
def send_email_basic(sender, receiver, email_subject):port = 465  # For SSLsmtp_server = "smtp.gmail.com"sender_email = sender  # Enter your addressreceiver_email = receiver  # Enter receiver addresspassword = 'env variable' # Enter your gmail passwordemail_html = """<html>  <body>    <p>Hello!</p>    <p>Add any text you'd like to the body of the e-mail here!</p>   <br>  </body></html>"""message = MIMEMultipart("multipart")# Turn these into plain/html MIMEText objectspart2 = MIMEText(email_html, "html")# Add HTML/plain-text parts to MIMEMultipart message# The email client will try to render the last part firstmessage.attach(part2)message["Subject"] = email_subjectmessage["From"] = sender_email## iterating through the receiver listfor i, val in enumerate(receiver):message["To"] = valcontext = ssl.create_default_context()with smtplib.SMTP_SSL(smtp_server, port, context=context) as server:server.login(sender_email, password)server.sendmail(sender_email, receiver_email, message.as_string())
In this function, we are doing a few things:
We are identifying the SSL port
Identifying the SMTP server
Storing the sender and recipient addresses in variables, as well as the sender’s e-mail password.
Creating a text string that has the custom html for our e-mail body and saving it to a variable
Identifying the e-mail as a “multipart” e-mail, and attaching the body text to our e-mail message
Attaching a “subject” and “from” parameters to our message
Iterating through our receiver list (in case of multiple recipients for the e-mail) to add to our “to” parameter of our message
Connecting to our server and sending the e-mail with the message parameters for “to”, “from”, “subject” and “message”.
Although there appears to be a lot to this, the function call is pretty straightforward. For more information about some of the e-mail jargon above, feel free to read up on this process here: https://realpython.com/python-send-email/ (same article as above for setting up your Gmail account as a proper sender).
Next, we can declare our next function:
def init_email():sender = "yourgmailaddress@gmail.com" # add the sender gmail address hererecipients = ["email address 1", "email address 2"] # add your e-mail recipients heresubject = "Subject Line" # add the subject of the e-mail you'd like heresend_email_basic(sender, recipients, subject)
This function, init_email(), will be used by our Python Operator to trigger our e-mail in a bit. Here is a breakdown of what’s happening above:
We are saving our sender address (which is the Gmail address you’ll use to send the e-mails from) in a sender variable
We are saving our recipient addresses as a list in a recipients variable. Here, since we have set up a way to iterate through this list in our previous function, we can just list out our e-mail addresses in the list here.
We are creating the text for our subject line and saving it to a subject variable
Finally, we are going to add these variables as parameters to our initial send_email_basic() function
Finally, we can set up our Airflow syntax to initialize this DAG and add the parameters needed:
default_args = {  'owner': 'airflow',  'depends_on_past': False,  'start_date': 'trigger date',  'email': ['your Airflow email'],  'email_on_failure': True,  'email_on_retry': True,  'retries': 0}dag = DAG('email_automation',           default_args=default_args,           dagrun_timeout=timedelta(minutes=5),         # schedule_interval = '0 13 * * 1', # if you want to set up an automation schedule you can do that here           catchup=False)t1 = PythonOperator(     task_id='send_email',     python_callable=init_email,     dag=dag)t1
Here we are doing the following:
Creating our default arguments for our DAG, similar to the Airflow documentation.
Creating our DAG(), naming it email_automation .
Using the PythonOperator() to create a task, call our init_email python function.
Assigning our task as t1 .
Once this is ready to go, your DAG file should be all set! By running:
airflow init db
You should be able to now see a new dag loaded in as email_automation.
If you’ve made it to this point, congratulations! With this sample DAG, you can now do a lot of different things:
If you’d like to truly automate this to send an e-mail to recipients at certain intervals (i.e. every day, week, etc.), you can uncomment the “schedule_interval” syntax in the DAG from above. More information about how to create syntax for this piece can be found here.
Additionally, you can edit the e-mail body message by adjusting the “email_html” variable text.
As mentioned in my opening, I’ve used these functions from this DAG as a task in a larger pipeline. For instance, I’ve added an e-mail task at the end of a data pull to send a notice to recipients that new data has been loaded, a new file has been created, etc. Given you can customize the e-mail body language, you can add links to your final data sources, as well as any other important information about the pipeline instance itself.
To close, Apache Airflow helps Data Engineers automate a lot of things! I hope this walkthrough was a useful demonstration of how to effectively use SMTP libraries and Python to make e-mail automation an easy task!
Thanks for reading! If you liked this article, feel free to head over to my profile for other pieces, where I discuss other Data Engineering, Data Science, and Data Visualization projects I’ve worked on.
Additionally, please check out my portfolio at: zach-alexander.com.