The Social Cost of Scaling Software — and Living With It.
By: John Contad
Tags: Management, Leadership, Engineering, Software Development

“A system of cells interlinked within cells interlinked within cells interlinked within one stem, dreadfully distinct…”

—Baseline test, Blade Runner 2049 (2017)

What’s the problem?

If you work within a fairly large software development organization (say, >150), your organisational structure might have different supergroups, each composed of multiple teams. It might look like this:

I blame Spotify.

Each supergroup would have a different mission and priorities, which branch off to multiple individual teams, each with a different mission and priority, composed of individual developers with their own internal rubric of missions and priorities —

You get the idea.

If you’re particularly lucky (or unlucky), the group might be developing to a single product — and in some cases, a single stack. This is hard, for reasons that are very fundamentally human. I’m going to spend some time explaining why.

A Thought Experiment.

Let’s say you (Developer Red) joined a team at the same time as another developer (Developer Blue), so you can work on the same things for say…5 years. You shared the same incidents, ran through the same sprints, did the same kickoffs, had the same leaders.

I would submit that at the end of that period, you and Developer Blue would develop similar priorities and sensibilities over time. Not completely of course, but there would be things that you both would hold as important.

This sounds glib, but let’s say we flip to the opposite. We made a copy of you — with everything that you are, everything you hate, and everything you hold important. But we split you off to two different teams for the same time period.

Off-screen: John Locke and Immanuel Kant screaming at my explanation.

One version of you went through a team that had a lot of crunch, had leadership that indexed towards customer value as important. The other had a bit of flex and time to index for quality. Different projects, different leaders, different teams.

I would submit that at the end of that process, you would end up with two very different developers.

I guess what I’m trying to say is, our sensibilities and the things that we hold important are largely (and in some thought circles, completely) defined by our immediate environment, the constraints that we live in, and the people who are around it.

Why this makes things hard.

Unfortunately, a very fundamental aspect of software delivery is being in discussions with other people who have those different fundamental priorities and sensibilities.

90% of the job.

Remember: each tribe would have their own priority, their own mission — so these conversations would be happening all the time.

The problem being human makes it vulnerable to human constraints. For some people, it takes a large amount of effort to empathise, reconcile, and understand another viewpoint. Whether it’s due to an avoidance of conflict, or the expense of emotional labor, people will end up creating groups who hold the same sensibilities. We like talking to people who aren’t a lot of work.

“They’re so rooted in their opinion, I just can’t deal.”

And if we’re not careful, we might accidentally end up with hard-drawn bifurcations of opinion (and perhaps even social groups). And if we’re really not careful, the social partitions that we have end up accidentally representing themselves in the software that we create. How many times have we seen different organisational groups with fundamentally different stacks or practices?

Conway’s Law basically, explained in a roundabout way.

I use the word accidentally deliberately; there is a choice that software development organisations must consciously make when they get to a size that enables the above social problems. We see this all the time in different frames (Standards vs. Autonomy, Consistency vs. Enablement, Leverage vs. Independence) — which is essentially a question of: How much do we have to be aligned?

I don’t think there’s a “right” choice here — only a set of tradeoffs that would apply themselves differently in different organisational contexts. However, it is important that an active choice is made as it implies an understanding of the tradeoffs.

After all, if we don’t make a decision, then a decision will be made for us by circumstance.

A caveat.

Briefly: This is just one of the possible failure modes of large software organisations — one that I care about deeply. There’s a lot more, covered much more in depth, but we’re focusing on just this for now.