Your Analysts Are Not a Search Engine
By: John Nardone
Tags: Consulting, Data Science, Business, Analytics, Strategy

Your Analysts Are Not a Search Engine

How to frame the problem and stop collecting shiny rocks

This article is a part of my series on Smart Analytics, targeted at executive leadership and data analysts. Check out my page to see the other topics.

Are you a business leader who has recently invested in an analytics capability, but does not feel like you’re experiencing the returns you expected? Are you an analyst who feels like no matter how good your work is, no one ever seems to do anything with it? Do you ask questions of your data on a regular basis, only to receive slides full of charts and boredom instead of actionable insight? Are you constantly asked to provide answers to questions by “EOD” as if you were a search engine for data?

If you’ve experienced any of these symptoms, you could be suffering from a disease called “Shiny Rock” analytics. It is characterized by high investments in time and effort, massive amounts of analytical documentation output, and minimal or negative value reflected on the balance sheet. The result is a collection of “shiny rocks,” or mildly interesting tidbits that don’t actually lead to tangible value.

Luckily, there is a cure for this malady. There are a discrete set of components that must be defined at the outset to give your analytics the best chance of driving value. If any of these components are not effectively established, your analysis is likely to at best yield nothing more than shiny rocks, or at worst — produce a result that seems right but is actually flawed and leads to negative side effects.

Ed See, a partner at McKinsey and a marketing analytics pioneer, explains the importance of having a plan before launching into analysis.

“To be truly impactful, an analysis has to be structured to answer a

question, and ideally, provide a path to action. Exploratory

analytics are important, but the power and value in analytics is

ensuring that the insights answer a question, and lead to driving a

business goal. Without that plan to impact, an analytics team can

quickly become just a ‘fun fact factory.’”

What follows is a 5-step planning framework used by strategy consultants such as Mr. See, as well as seasoned in-house analytics teams. I’ve witnessed firsthand how dramatic the impact is when this process is executed successfully. Your business stakeholders will thank you for how efficiently value is created. Your analysts will thank you for the clarity of direction. And your competition will fear your seemingly clairvoyant decision-making.

The Smart Analytics Pyramid of Success

Step 1: Define the Strategic Decision

Conducting an analysis is like building a house, and the business decision you’re trying to make is the foundation. Once the foundation is set and the work begins, you can’t change it without ripping down everything else that was built on top of it.

I am always shocked by how often this step is glossed over or skipped altogether when beginning any analytical initiative. The point of analysis is to help inform decisions, so take the time to decide what that decision is.

Then, document it. Make sure all stakeholders are aligned and in agreement. This will become your North Star — everything that comes after must tie to the strategic decision.

Lastly, make sure that your stakeholders internalize this: one decision = one analysis. Don’t try to boil the ocean. Keep a backlog of lower priority initiatives and tackle them one at a time.

Step 2: Converge on Actionable Questions

Once you’ve settled on the decision you’re trying to make, you need to identify the pieces of information you can use to inform that strategy. These questions have to be actionable — if you knew the answer to the question, what could you do with it?

Your analysts aren’t a search engine — every time you ask one to provide an answer, it will likely require hours of their time. You wouldn’t spend money on technology that you aren’t going to use, don’t seek answers that you can’t act on.

You can test if a desired piece of information is actionable if you can put it in the form of “if we knew (blank) then we could do (blank).” For example, “If we knew which of our customers also shop at our competitor, then we can target them with tailored messaging.”

By defining questions this way, you’ll ensure that your analysis is guided by the desired output, and you’ll avoid collecting shiny rocks.

It can also be helpful to categorize each question as “essential” or “nice to have” in the context of informing on the strategic decision. Later on in this process, you may realize that there are gaps in the data you have access to— having your questions prioritized will help you decide how much effort to invest in filling those gaps.

Step 3: Attach Key Metrics

Up to this point, every element of the framework has been driven by the business. Now, you must start to transition into analysis decisions. This is the part where having a good analytics translator becomes really helpful. As the name would suggest, this is someone who is fluent on both the business side and analytics side, and can serve as the liaison between the two groups.

Leverage your translator to help define the metrics you will use to answer each actionable question. By definition, a metric is something you can measure. An example of a metric we might define for our earlier question could be “total dollars spent in 2019 by our customers at our competitor.”

Step 4: Choose an Analysis

By now, you’ll be fully on the analytics side of the equation. For each of your metrics, you need to decide how you’ll leverage them to answer the question they tie to.

Is this something that requires a descriptive solution, such as a dashboard that can be monitored over time? Do you need to build a model? Are you trying to forecast the future? Is this a one-off effort, or is there value in automation? What constraints, thresholds, and targets can you set to ensure your analysis provides useable results?

The expertise of your translator and analysts will help define what the solution will look like. There are countless articles about the analytics value chain, so I won’t go into that here. The key point is that your analytics experts are the people that will be tasked with executing this solution — make sure they’re involved in deciding what that solution will be.

Step 5: Identify Data Sources

The last component of any successful analytics endeavor is identifying the data sources you will use to power your analyses. I’ve seen strong analytics teams identify the data sources right after defining the questions to answer. More often than not, this approach results in the questions getting tailored to the data, not the other way around. The scope of the analysis gets restricted to what you think you can answer, and not what you should try to answer.

By identifying your data sources at the very end of the analytics planning process, the gaps in available information will become clear. You can make decisions about whether to invest in new systems or third-party data, and if you tagged your questions based on priority, then you’ll have a better understanding of how much to invest.

Here’s the thing about frameworks -

They only work if you use them. Business leaders need to be disciplined in defining their decisions and actionable questions up front, and doing what they can to avoid changing them mid-analysis. You need to make sure your stakeholders are aligned, and that if additional requirements evolve, they fit into the agreed-upon structure. You need to involve your analysts in the conversation, and trust their expertise in defining the methodology and identifying the data sources.

Analysts need to speak up when being asked to spend time working on something without understanding what the potential impact is. If you’re getting asked to answer questions with no context, ask for it! Demand to see where the question fits into the framework, and clearly articulate the level of effort associated with finding an answer.

Most analyses fall short due to a lack of structure from the outset. The 5-step approach to planning can provide that structure and increase the likelihood that your analytics will create value. You’ll ensure that business decisions drive the analysis, your stakeholders are in alignment, and if all goes according to plan — your analysts will stop feeling like a search engine.