Well That Escalated Quickly — Building Better Products with Escalation
By: Saumil Mehta
Tags: Product Management, Startup, Software Development

Impasse is common in software development — escalations should be too

After the basic pleasantries were concluded, I sat across from the seasoned Amazon Eng leader and asked him one of my favorite questions — “After 3 months at Square, what have you found to be most surprising about how we work compared to Amazon?”. Like all San Francisco tech types, I retain a healthy level of obsession with the Amazon way and am always looking to learn from those with hard-won tribal knowledge. I expected a well-worn answer about Amazon’s frugality, or its ingrained long-term orientation, or its relentless focus on working backwards from the customer. Been there heard that.

What I heard, instead, was something I had felt but never been able to articulate with clarity — he said, “The thing that surprises me most is how little we escalate around here. Escalations were everywhere at Amazon and just a way of life. They didn’t mean anything negative to us, we would just write up an escalation to various teams and leads and explain tradeoffs”.

Building and shipping software with predictability and velocity is hard. It is hard in small startups but also, in different ways, in fast-growing larger companies with hundreds of semi-autonomous teams with their own roadmaps. While the 2 environments have several challenges in common, larger companies with hundreds of teams find themselves continually challenged to ship work that crosses team boundaries — that is, projects that require 2 or more teams to problem-solve together, design together, coordinate priorities and timelines. In my years at Square, we’ve found this to be somewhat difficult nearly 100% of the times even in the simplest of cases — even when exactly 2 teams are involved and both directly report up to me. Of course, the more teams we add and the more organizational distance we add (e.g. different General Managers), the higher the difficulty level. To express this visually for those such inclined:

Yes I know the axes aren’t labeled properly…but you get the idea

Even if all teams are strict adherents to the no-asshole rule, the difficulty comes down to simple differences of opinion in 1/ prioritization (e.g. P0 in one team, P2 in another) 2/ Solution Design (technical, user experience or features) 3/ sequencing of work in the right order and 4/ ownership and decision-making authority.

While there are entire content marketing playbooks out there on how to run teams more effectively at scale, one of the easiest and most effective things you can do as a Product, Design or Eng Manager is to get better at escalating more effectively. To be precise, an Escalation is defined as the act of resolving an impasse across 2 or more teams by quickly and recursively traveling the org chart to present trade-offs and choices until a disagree-and-commit decision is made.

Project teams get together to escalate until the final point

This sounds incredibly Captain Obvious in theory until you root cause why teams fail at doing this on time or at all in practice…all the time. Why do teams fail? Because they are populated by humans with human tendencies — the desire to get along, the desire to not upset the apple cart, the desire to not bother their manager because they are “too busy”, the desire to avoid what sounds like a confrontation (i.e. “that escalated quickly” in the famous Ron Burgundy formulation).

Square has gotten better at this over the years but we are still working at it, mainly by being vocal as a senior leadership team to the rest of our teams and by continuing to iterate on our planning processes. If you find yourself in a similar situation, here are a few best practices to keep in mind:

Create the right permission structure: Leaders must vocally create the permission structure — since the default behaviors are rooted in deep-seated human tendencies, alterations to behaviors require vocal advocacy from senior leadership — it’s OK, it’s healthy, you’re not doing anything wrong, you should be doing this more often. Embracing this supposedly contrarian position helps. As someone else from Amazon said about this, “Escalation success [in a company] is conditional on the fact that everyone needs to believe that escalations are not negative. It’s a prisoner’s dilemma game theory problem…if you believe that an escalation will not be perceived as a negative, then my best response is to escalate and vice versa”. Escalate Respectfully and Openly: Even if you have the right permission structure, the act of deciding to escalate can be fraught with anxiety about looming conflict. This is incredibly natural. Coach your teams to have respectful, polite but direct conversations with partner teams about the need to escalate and a collaborative process by which to escalate. Coach your teams to double-check that they are at an impasse that cannot be resolved before they escalate. Write SPA Docs: Originally created and popularized by everyone’s friend and advisor Gokul Rajaram, the Square SPADE framework is now templatized and available everywhere. While technically about decisions, it turns out to be just as helpful with escalations — our standard practice is to ship what we call a SPA doc. In this case, Setting, People and Alternatives are clearly documented but Decisions and Explanations are left blank until the impasse gets resolved. This helps the final call maker weigh options easily and clearly. Frequently, teams on different sides of an impasse are asked to write their own alternatives to faithfully and fully represent options — no shading! Force Recursive Traversal: It is bad form to blindside the management chain in a different org, or even in your own org, by going straight to the senior-most executive and skipping several layers of managers along the way. Since healthy organizations push decision-making down through the org, unfortunately deadlocks need to be resolved by escalating up the chart one step at a time. On the flip side, if you are the final deciding senior executive — it is also bad form to make a call when this step-by-step escalation is not followed. The game only works when all teams play by the rules. Explain Why: If you are the final decision-maker making a tricky call with meaningful downstream consequences — you must explain why. Outlining a line of reasoning with clarity helps teams calibrate their own judgment and serves an important contextual training function.

Finally, if you want to minimize the number of escalations through a year — plan better. Annual and quarterly planning is pooh-poohed by those who think planning is for staid, old-line big companies — been there — but planning reasonably well can concentrate the time window during which many escalations happen at once. For example, the Square annual planning process lasts several weeks in Q4 and many find the process exhausting and, dare I say, a bit bureaucratic — but it also produces dozens of consequential escalations and decisions in a short period of time and saves our skin through the next year. Whatever we escalate after that is stuff that was probably impossible to foresee. In the absence of a good planning process, we’d simply Snack and never Eat.

That’s it! As with all things at scale, this is unfortunately easier said than done in practice. But with the right leadership and the right permission structure, escalations can be a useful decision-making tool and a key source of velocity and predictability. Just don’t let them get out of hand like, you know, Ron Burgundy.