An Error Alerting System of One’s Own
By: Lara Aasem
Tags: Open Source, Slack, Software Engineering, Error Alerting, Errbit

An Error Alerting System of One’s Own

“Software engineering is strewn with the wreckage of those who have relied beyond reason solely on the error alerting configurations of others.” — Virginia Woolf, paraphrased Lara Aasem Follow Jan 23, 2019 · 7 min read

Picture this. It’s 5PM on a Saturday. By some twist of fate you do not yet completely understand, you’ve ended up both on call and committed to a family dinner at this ungodly hour of the Sabbath. You think it’ll be fine; you’ll just keep your laptop handy and periodically check your phone for error alerts until you safely ride out your on-call time slot. Right?

Fast forward to 6:07PM. You’re sprawled on the floor of your grandma’s bedroom, both laptop and phone open to your Slack alert channels and a shell on a production machine running, all on mobile data. Some production issue has vital services down and alerts that all look the same rolling in every millisecond. Your product manager is going crazy, and you yourself can barely breathe, let alone figure out what’s wrong and fix it. You start ruing the day you went into software engineering and wondering if pyramid schemes really are the scam they very obviously are.

Alerts, alerts, alerts

Before you call your friend who pitched you the latest Ponzi, know it doesn’t have to be that way. As hyperbolic as the above hypothetical situation is, it captures the essence of a very real software engineering problem: on-call fatigue. Yes, it’s great that you have an error monitoring system in place and that you’ve linked it to your workplace messaging app, but constantly seeing alerts that all look the same coming in at a huge frequency is just overwhelming. As time goes on, stress and burnout build up within the engineering team, production environment deviance becomes normalized, and issues are either unresolved or resolved very inefficiently.

This is why we decided to personalize our error alerting system. We narrowed the problem down into three major sub-problems:

alerts weren’t being routed to the concerned team members, there was no way to differentiate between them at a glance, and they had too little helpful information.

We use Errbit, an open-source Airbrake-API-compliant error catcher that, like our own BulkWhiz API, is written in Ruby-on-Rails. Errbit supports a number of notification services (and, as per the essence of this post, you can always extend it to support even more), but we use Slack so that’s what I’ll focus on here.

To get started, we forked the Errbit repo on GitHub, created a Docker image and pushed it to DockerHub, initialized a test Errbit Kubernetes deployment pointing to that image and routed staging alerts to it, and we were good to go!

Don’t @ me — but actually @ me

What if you didn’t have to worry about every single alert ever but only the ones you were mentioned in? No more keeping tabs on the alerts channel for a good half hour after pushing production changes or analyzing a backtrace to figure out which fellow team member to tag. Our first thought was: Create a routing system within Errbit that gets git blame information from GitHub's GraphQL API. Map the returned JSON to the in-app files and line numbers in the error's backtrace and you have the GitHub usernames of the code authors. Map those to their corresponding Slack user IDs which you can get from the Slack API, either by getting them from Slack's API method testing interface and passing them in an environmental variable to your Errbit deployment or, cleaner and more robust to changes, calling Slack's API from inside your Errbit code.

One of our first alerts with code author mentions.

Now that’s all good and well, but “whodunnit” isn’t necessarily “whogottadealwithit.” As such, we implemented a force-assignment system, mapping certain error classes to the Slack user IDs of the responsible team members and checking that hash first before sending the git blame request. We thus had to create some custom error classes from our BulkWhiz API codebase to be able to raise those force-assigned errors where needed. Force assignment also proved useful in situations where the code author was no longer part of the team, or where the error at hand was a broader issue that more than one team member needed to deal with. In those cases, we decided to tag Slack user groups instead of individual users. (Pro tip: In code, individual Slack user mentions ( "<@#{slack_user_id}>" ) work a bit differently from Slack user group mentions:

"<!subteam^#{slack_user_id}|#{slack_user_name}>" .)

Triage and error

To make processing alerts even faster, we needed to begin differentiating between their levels of severity at a glance. Some alerts are actually uncaught exceptions, whereas others are notifications we manually raise from the BulkWhiz API codebase to trigger certain actions, both automated and human. Using color coding and emojis, we’re now able to visually differentiate between those two alert types at first sight.