Too Much Data, Not Enough Insight
By: James J. Ward
Tags: Data Science, Data, Business

It’s a strange phenomenon of modern business that we are absolutely awash in data but, frequently, it seems like the additional information doesn’t lead to insights or meaningful tools. For many enterprises — especially small businesses — market necessity makes it seem like a move to a comprehensive system with data-analytics is the only way to survive. But once they get Shopify or SalesForce or whatever SaaS or PaaS they fancy, how many companies actually start gleaning useful information at all, to say nothing of creating valuable data products as a result? Given that nearly half of businesses don’t even know where their data is stored, the answer is “not nearly enough.”

That’s not exactly what I said, Paul, but fine.

Drinking from a Fire Hose

Some of this comes down to the amount of information at play. Much of the time, when discussing the amount of data moving through the Internet, you hear about how there are more bytes online than there are stars in the Universe, or that in ten years we’ll produce more data than is stored in the Library of Congress every hour blah blah blah. We get it: there’s a lot of data.

Why, amid an absolute glut of data, are businesses finding it difficult to meaningfully deploy it for new, valuable purposes? A few reasons. That very volume of data can be overwhelming, and creates a sense of impossibility to manage. Consider the fact that, in the marketing industry — which is completely reliant on data both to devise its strategies and prove its worth — there was already a widespread sense of being overwhelmed by data, and that was five years ago. The relentless pace of data inflow has only accelerated since then. If marketers, whose lifeblood is data, can’t keep up, how can anyone else?

These stats and surveys hide the important facts while still pointing to them. Saying that there is a huge amount of data (“44 zettabytes!”) is meaningless: humans are cognitively incapable of processing what that kind of number means. Think about it this way: try to imagine 100,000 people in a football stadium. Not impossible, right? Now imagine a million different stadiums, each holding 100,000 people. Harder? Now imagine 5,000 planets, each with a million stadiums on them. Replace “people” with “bytes” and you’ve reach the amount of data Americans produce online — every minute.

I won’t lie, that paragraph took me 45 minutes to write.

What Are We Looking For

“But with all of that data,” the thinking goes, “there must be value and we should grab it!” It’s not unreasonable to assume so, but when you analyze it carefully, how much of that information is valuable to you? How does it tie to revenue? Who can make use of it in the immediate term, without needing to construct a massive machine learning tool?

Often, the problem is that we’re very focused on answers when we should instead be focused on questions. “Data” isn’t an answer, it’s just another way of saying “stuff.” When you look at the data you have internally or as a result of a partnership with another company, saying that you want to find a way to utilize data sounds profound, but when you come down to it, what you’re really saying is “I’m sure there’s a way to use this stuff.” Less professional, but no less accurate.

Asking the right question, then, is substantially more important than assuming there is an answer. But vague, generalized questions are just as useless as vast stores of data with no purpose. The best course is to ask yourself questions that don’t presuppose the answer, but instead cause you to explore the data you have or realize where the data you have is lacking.

Bad Questions, Good Questions

Consider the kind of inquiries that you would start, and what kind of data you would need, based off of questions like these:

What do customers want?

How much more money can we make by using data?

What are the best ways to change our marketing plans?

Which of our employees is the most effective?

You could spent months trying to figure out what these questions even mean. “What do customers want?” What, for Christmas? But this is exactly the kind of amorphous, vague inquiry you’ll get if you don’t push for concrete, real-world questions about data. Compare those questions to these:

What gaps do we consistently find in our client acquisition process and Know-Your-Client efforts?

What order fulfilment metrics are we not meeting, and why?

Which of our employees has the highest conversion rate when tying attendance at conferences to closing customer relationships?

Is there a correlation between our marketing spend and sales increases?

Are we finding a greater delay in resolving invoices since our migration to a new vendor management platform?

These questions are so much better largely because they can be answered. “Who is our most effective employee” has no measurement metric, where “who gets us the most business from going to conferences” is an answerable, measurable, verifiable question that would be super valuable if conferences existed anymore. Again, the key is to focus on finding the right questions that data can answer, not the other way around. If you do, you can wade through the “data lake” and find what’s really meaningful. Most importantly, you avoid the mistake of trying to do everything with your data and wind up doing nothing at all.