by mrandish | 2025-04-11 18:41:43+00:00
> you will probably add in some user analytics service, and use the insights from that analysis to inform future development. However, that analysis will aggregate its results over all your users, and won't pick out the enthusiasts, who will shape discourse and public opinion about your service. Consequently, your results will be dominated by people who don't really have an opinion, and just take whatever they're given.

This is so spot on. I was a long-time serial entrepreneur who spent a couple decades across three successful startups discovering, shipping and growing new categories of tech products primarily for consumer, prosumer and hobbyists. Then I sold my last startup to a very large F500 silicon valley tech leader and ended up a senior product exec there. While there were a lot of positives like more mature engineering processes, testing and devops as a discipline, the exact issue you describe was a nightmare of product-damaging mistakes I called "analytics abuse." In my startups I valued having increasingly robust analytics over the years. In part because they helped increase my overall understanding of usage but mostly because they provoked good questions to explore. That exploration happened naturally because as the "product guy / founder" I never stopped spending a lot of time with our most passionate, opinionated, thought-leading customers. Over years of iteration I'd learned how to engage deeply and listen carefully to input from these customers. This involved interpreting, filtering and curating the mess of divergent personal preferences and pet feature ideas to tease out the more actionable product signals that could increase broad usage, adoption and passion around our products. I'd then bring those curated signals back to the product teams for evaluation and prioritization.

At BigCo they were diligent about meeting with customers, in fact they had entire processes around it, but their rigorous structures and meeting agendas often got in the way of just directly engaging and actively listening. Worse, the customer meetings the more senior product decision makers actually attended in person were mostly with the highest revenue customers. Junior PMs (and sometimes new grads) were delegated to meeting with the broader base of customers and filing reports. Those reports were then aggregated by ever-helpful program managers into tables of data and, eventually, slides - losing all nuance and any ability to spot an emerging outlier signal and tug on that thread to see where it goes.

I tried to convince everyone that we were missing important customer signals, especially from our smartest, most committed users. Being only one level removed from the CEO and quite credible based on prior success, I was definitely heard and most people agreed there was something being lost but no one could suggest a way to modify what we were doing that could scale across dozens of major products and hundreds of product managers, designers, execs and other stakeholders. In my experience, this general problem is why large companies, even the most well-run, successful ones full of smart people trying their best, end up gradually nerfing the deeper appeal in their own products. Frustratingly, almost every small, single step in that long slide pushes some short-term metric upward but the cumulative effect is the product loses another tiny piece of the soul that made our most evangelistic, thought-leading customers love the product and promote it widely. Ultimately, I ended up constantly arguing we should forego the uplift from some small, easy-to-prove, metric-chasing change to preserve some cumulative whole most people in the org weren't fully convinced even existed. It was exhausting. And there's no fighting the tide of people incentivized on narrow KPIs come bonus season.

I'm sorry to report I never found a solution to this problem, despite my best efforts over several years. I think it's just fundamental. Eventually I just told friends, "It's a genetic problem that's, sadly, endemic to the breed" (the 'breed' being well-run, very large tech companies with the smartest product people HR can hire at sufficient scale). Even if I was anointed CEO, given the size of the product matrix, I could only have personally driven a handful of products. I do think codifying premises and principles from the CEO level can help but it still gets diluted as the number of products, people and processes scales.