The Agile dilemma: is it always right?
By: Josh Dormont
Tags: Product Management, UX, Agile, UX Research, Lean Startup

It seems like every week someone is either doing a full out take-down of Agile or preaching something akin to “You’re doing it wrong!” I get it. Peak in management magazines like Harvard Business Review and Agile reads like the next Innovator’s Dilemma — even for non-tech companies. Yet just about everyone agrees that Agile’s principles get a lot of things right.

Where things go wrong is often in the specific implementation: in the adherence to a particular model that doesn’t fit, a lack of leadership buy-in, or something else altogether. At the end of the day does it matter? Does it matter what big principle the team is operating under if users like the product, the team is happy, and the stakeholders are supportive?

Does it matter what big principle the team is operating under if users like the product, the team is happy, and the stakeholders are supportive?

I’m going to try to convince you the answer isn’t just a “it depends,” but is a solid yes. First, a story.

Not too Agile. Not too Waterfall. Finding the Space Between.

Imagine for a moment that you’re a product manager who has received a set of high level priorities from senior leadership. These priorities — somewhere between requirements and problems to solve — must be met by a certain date, must include certain sets of data, and must address the needs of a specific audience. Initially this sounds like the kind of thing that Marty Cagan would call a “High Integrity Commitment” that a product team must be able to take on. But there’s a catch — there are no plans to release the product until all the priorities and other features are designed, tested, and built. It’s an eight month window for the product team to do their work. Sounds like Waterfall, right?

Not quite. The team has plans to design and test a series of increasingly high-fidelity prototypes with different users. They work in sprints with all the normal activities you’d expect to see. The team is larger than average (about 15), and stacked more with designers, UX researchers, and policy experts than developers. That’s just how things fly at the org. They incorporated earlier user research into their selection of some of the other features they want to build and have a few assumptions they want to test. Sounds pretty Agile, right?

In general, people on the team are excited to be working on it. They recognize that their work is driven partially by org priorities and stakeholder interests and partially by user research that they’ve done. They’re excited to be working with new people and honing skills they’re looking to develop.

We come back to the question: does it matter if this team is running a hybrid model?

Agile matters because the solution is probably wrong

Great user research can help you avoid big errors. Testing with prototypes will get you feedback that will increase the chance that your hypotheses for how to meet your users’ needs are right.

But once you release, you’re likely to quickly realize where you were wrong. Where some assumptions were just off or where what people said they would do was very different from what they actually did. Where maybe you overestimated their desire to switch to something new. Where even with the ‘right’ data, it wasn’t necessarily the right data for the situation and practice.

Maybe you get lucky and the majority of the features you created fulfill the needs you were addressing — but how does that land for the stakeholders who needed to see it all, for one specific scenario and who now need an alternative?

To zoom back out for a moment, Agile isn’t just about iterative design and user research. That work is hugely important and often undervalued. But the difference between user research and live experiments are night and day. This is just human behavior, and our biases and irrational behaviors are predictable but not always preventable.

Discovery means Experimentation

When I talk with product managers, project managers, scrum masters, data analysts, and data scientists, just about everyone has a different definition of what an experiment is. I find this funny because if you talk to researchers in just about any commercial or academic area, they basically all have the same definition: a set of unknowns you test starting with a hypothesis. The other thing those researchers share? They assume they’re wrong and expect to test things many, many times.

So how can we as people doing product things help channel the scientific researcher way?

Do your research. I’m not advocating for skipping user research, just for understanding what types of assumptions it is best posed to address. Test with the right level of prototype. As I’ve written about before, prototyping is not a linear process from low- to high. Some tests need a live-data prototype that’s going to require developer time. Other tests could be just fine with design-only methods. Release to a limited audience. Run experiments in production with a limited group of users to get both qualitative and quantitative feedback. There is simply no substitute for this step and the best companies are doing it by the hour. (Seriously, there are hundreds of simultaneous experiments being run when you log on to site like Amazon, Netflix, or Google.) Share what you’re learning. The only way you’re going to build trust with your users and stakeholders is if you share what is and isn’t working. If you feel like you need a long window to do your work, there might a deep trust issue that needs to be addressed first. Break the work up into multiple empowered product teams. Maybe this should come earlier, but one way to create space to test and pivot within a product is for teams to have the autonomy and flexibility to do so. Having just one team (one PM, one group of designers, one team of developers) creates too many dependencies to disentangle the work efficiently. Maybe they’ll have better coherence and coordination, but those problems can be solved in other ways. Listen to the data, but follow up in person. Data from your experiments are hugely important in helping you understand where your hypotheses were right and wrong. But they don’t help you with the ‘why’? Look at your analytic tools, your logs, and develop a new set of questions you want to ask users in your test group.

The Agile Effect on Design and Engineering Culture

There’s another subtler, but I’d argue equally important reason for an Agile approach: culture. While the research on this is still developing, a group of Lean and DevOps experts pulled together surveys, interviews, and other data to lay out a set of findings on what the strongest dev teams shared in common in a book Accelerate. The result?

Teams that practiced more elements of Continuous Delivery, Continuous Improvement, and broader Agile practices outperformed others, stayed together longer, built better products, and were happier.

Maybe you get lucky. Maybe those things matter less to your PMs, DBAs, engineers, and designers. But in the long run, that’s not what the research suggests.