Quick & Easy Alerting for Apache Airflow
By: Raouf Aghrout
Tags: Airflow, Open Source, DevOps, Opsgenie, Data Engineering

A Task Doomed to Fail

As our starting point let’s define a DAG with a single task that runs a simple BashOperator.

Our eternally failing task.

We’ve made sure this operator always fails by simply calling exit 1 so we can test what we want to do off the back of the failure.

The Power of BaseOperator

All operators in Airflow inherit from the BaseOperator class, which means for any operator we’re using, including our BashOperator, we’ll have access to the parameters available in this class. The BaseOperator parameter we’ll be taking advantage of specifically is:

on_failure_callback (callable) — a function to be called when a task instance of this task fails. a context dictionary is passed as a single parameter to this function. Context contains references to related objects to the task instance and is documented under the macros section of the API.

This can be configured to call a custom function in the event of our operator failing, which is exactly what we’re going to do next with the BashOperator we defined earlier.

Our task now does something when it fails, thanks to on_failure_callback.

The do_something function that’s triggered when our task fails currently isn’t defined, but we’ll get there. The context dictionary that’s mentioned in the documentation is also key for us, this will allows us to get metadata about the instance of the task that has just failed. Later on we’ll use this context dictionary to customise our alerts, making them informative and user friendly.

Setting up our Connection

Airflow comes with a bunch of built in hooks, some of these are published by the Airflow team and others are community contributed. We’ll be using the community contributed OpsgenieAlertHook in our example, but you can find pretty much anything you need here. If you’re using a different alerting platform with an API you can achieve pretty much anything with HttpHook, which is also the parent class of OpsgenieAlertHook.

For this hook to work we need to configure a connection to Opsgenie, looking at the code for this hook it’s already setup to use the existing opsgenie_default connection that comes with Airflow. For simplicity I can edit this existing connection in the Airflow UI, this will save and encrypt it in our Airflow database. However, if you’re not keen on having this state lying around because you’re using the default in-memory Airflow database or haven’t encrypted your secrets, then you can also configure connections using environment variables or at run-time in your Airflow code. This is great if you want to grab the credentials from your secrets manager and the way I’d recommend doing this in a production environment. For now we’re just going to drop our Opsgenie API key into the password field and save this.

Configuring the Opsgenie connection in the Airflow UI.

Note: When you revisit this connection in the Airflow UI you won’t see anything in the password field, it isn’t even visible but obfuscated, however it should be persisted!

Leveraging the Hook

Let’s make our do_something function actually… do something, this will be our hook into our alerting platform. Let’s rename do_something to opsgenie_hook and import this functionality from a different Python file. This will allows us to build something generic and then easily re-use this code in all of our DAG definition files.

We’ve got a basic hook implemented.

So hook.execute() will, as configured in OpsgenieAlertHook by default, send a POST request to the default Opsgenie alerts API endpoint at https://api.opsgenie.com/v2/alerts. It’s able to authenticate against the Opsgenie API utilising the opsgenie_default connection that we just configured. We’re supplying the name of this connection as the argument to OpsgenieAlertHook constructor. As you’ve probably noticed, the execute function currently takes an empty object, but not for much longer, this will be our JSON payload.

Configuring & Customising our Alert

Remember we talked about the context dictionary earlier? You may have noticed that our opsgenie_hook is taking context as an argument, the contents of which we can use to build and customise our JSON payload.

Our finalised hook, complete with a custom message.

There’s a little bit going on here so let us unpack it piece by piece. Firstly we’re extracting some useful stuff from the context dictionary:

dag is the name of our DAG

is the name of our DAG task is the name of our Task

is the name of our Task ts is the timestamp of when our Task was scheduled to run

is the timestamp of when our Task was scheduled to run log_url is the location of the logs for the instance of this Task on our Airflow instance

These seemed the most useful but you can see all that’s available in the context dictionary in the TaskInstance class. We’re using dag, task and a slightly nicer formatted ts to construct a short informative message from which you can immediately discern what’s wrong. An example would be:

Airflow DAG eternal_failure, failed to run task_that_always_fails, scheduled at 2019–06–21 16:00.

We can also embed the log_url as metadata within the JSON payload itself, this makes our alert actionable, anybody responding to this alert can immediately head to the first port of call when troubleshooting an issue. We’re replacing localhost with the actual DNS of our airflow instance to avoid a manual step here too. The rest of this payload is pretty Opsgenie specific, but of course we’re routing the alert to the appropriate team, setting the appropriate priority for the alert, adding useful tags and so on. These are all things you’ll obviously want to tweak depending on your own implementation.

Our actionable alert arriving in the Opsgenie platform.

Opsgenie, like any alerting platform, also takes care of notifying us through various channels like Email, SMS and Slack. Having a concise, informative and actionable alert is critical when you’re on call. It’s also great to be able to interact with these alerts via Slack directly.

Interacting with our actionable alert in Slack.

Conclusion

We can all agree it’s important to know immediately when and why things aren’t working in production, especially in an era where we’re responsible for building, running and owning our complete stack end to end. The tasks you run as part of your Airflow DAGs are no exception to this, especially as it’s often likely you’re holding up some downstream workloads that can be business critical. There’s also no reason why setting up this kind of alerting should be a hassle either. Airflow delivers that ease with some really simple functionality that’s built in and trivial to leverage.

All of the Gists embedded here are available publicly and a concise final version of this example is available here.