Inside the Boiler Room
By: Cris Concepcion
Tags: DNC, Collaboration Tools, Technology, Engineering Mangement, Remote Working

Inside the Boiler Room

Collaborating as a distributed team

The DNC Tech Team has been working as a mostly remote team since fall of 2017. Colleagues are spread across 19 states and 4 time zones, and we have embraced collaborating with each other regardless of where we are. Much of our general toolbox and practices are covered in previous blog posts, but it’s worth exploring a specific practice that we use pretty regularly: the virtual boiler room.

What is a boiler room?

Most of us in the tech industry have been through something like this: A production outage occurs and you need to get a bunch of experts together to troubleshoot the issue. Or there’s a major promotion or deploy, and you need to keep people nearby and ready in case of trouble.

Many offices have an operations room or a network control center that becomes the epicenter for this kind of work. Often they have fancy displays of network maps and dashboard graphs to give everyone a visual overview of the health of their systems.

In political campaigns, an analogous pattern is the boiler room, where people are huddled together, keeping their eye on stats, reports, and trying to make some rapid fire decisions as field information comes in. How do you replicate this sort of deep, intense collaboration when you can’t be in the same room?

The virtual operations and boiler room

The basic approach is to use situational Slack channels. If an outage happens, create a channel specifically for engineers to go in and work with each other.

Instead of a variety of graphs and displays on an ops room wall, team members post links to an online dashboard or a screenshot of a graph with interesting data. It’s easy for these channels to get overwhelmed with people who want to know what’s going on. So, we also spin up an associated comms channel to post status reports and field questions, and coordinate user communications.

To allow engineers to focus on resolving the issue, we designate an incident manager, whose job is specifically to handle logistics and not get in the weeds of troubleshooting. We also keep a permanent outages channel to inform people when an outage starts, where it’s being discussed, and when it ends.

Use a designated channel to alert relevant team members to an outage

Once the event is passed, and when we have our postmortems and reviews, having the chat and decision record helps augment our often unreliable human memories and fill in details that we would otherwise forget.

This methodology is helpful in non-outage situations. During a major event like a multi-state primary, where there isn’t an outage, we’ll create channels for each state as well as one for the event itself. The state channels can be used for focused discussion, and then major issues are raised to the main event channel. This allows everyone to keep a broad view of what’s going on during the day, while additional, more specific channels give relevant team members the opportunity to learn more about a specific state or problem. Doing all of this in chat can also help others catch up on a fast-moving conversation. Rather than need to force someone to slow down and summarize things, newcomers can just scroll up to the most recent status update and read down from there.

Sometimes Slack isn’t enough. In those cases, we use video conferences to discuss things in person (virtually) and come to a decision quickly.

It’s always important to post back to a channel when the conference is over to record any decisions and next steps. If it isn’t written down, it doesn’t count.

Why is this important?

As more teams move to using Slack and video conferencing for getting things done, it’s easy to feel overwhelmed by all of the channels and all of the discussion.

You don’t need to read everything, you just need to read what matters to you.

By splitting channels into focus topics and also having one channel to take in summaries, you can reduce the cognitive load of navigating all of these conversations, and help discussions stay focused.

It may take a moment to adjust from a physical to virtual boiler room. However, for our team we’ve found it to be an effective way to work collaboratively with an entirely distributed team.