Is "Big Data" Just More Data? (2013)

EXECUTIVE SUMMARY |This column deals with Data, one of latest technology trends being hyped. It discusses what managers need to consider when evaluating whether or not to implement it at their companies. A brief synopsis of a new book introducing concept of signals versus noise is given, as well as some of lessons learned from efforts in downstream data-a long-standing industry initiative. The column recommends identifying a few key predictable signals from Big Data to focus on during implementation, and considering rest of data to be noise that adds no useful information towards improving decision making.Big Data is all rage these days as business community and technology suppliers talk over-and-over again about seemingly boundless opportunities from assembling a broad range of data to solve some of most pressing business problems. The Big Data concept is not new; it has always existed as more and more information has been generated throughout history at an exponentially growing rate-especially since dawn of computers. Today's era of Big Data is focused around vast amounts of information being generated by growing use of Internet in many aspects of our lives.I was a graduate student (in early days of computing) and took many business analysis courses. At that time, concept of Big Data embodied lots of data being generated (albeit on paper) and a significant portion of it was inaccurate, due to human errors. Since data was largely paper-based, one needed to be extremely judicious when assembling information. There was concept of garbage-in, garbage-out, wherein much of data might lead to faulty analysis and prediction. Picking right data streams to use was extremely important. The mantra that was followed was that not all data was information. Information supported improved decision-making, and not all data was useful for that. Generally, data is information if it is used within a decision-support system that has an underlying business model and principles imbedded in it.Since that time, while Big Data got even bigger, computing capabilities kept pace allowing it to be more easily and accurately assembled and analyzed. However, can all that data really provide significant enough information to improve business operations? Is Big Data just more data?Implementing Big Data is a big deal. Many practitioners are struggling with whether or not to implement it within their companies. They rightly question whether to begin this effort that requires enormous amounts of time, money, and other resources, while possibly distracting them from pursuing other opportunities that require less effort. The answer to their questions as to whether to go after Big Data is Yes. However, only if a Big Data implementation will sufficiently increase their ratio enough to yield significant benefits to cover large investments needed. What is a signal-to-noise ratio?THESIGNAL AND THE NOISEBOOKI just finished reading a book written by statistician, Nate Silver, titled The Signal and Noise: Why So Many Predictions Fail - But Some Don't. I recommend this book as an important one to read for anyone who needs to make decisions that impact future business operations; that is, just about any business manager, especially forecasters, planners and supply chain managers.In his introduction, Mr. Silver discusses the promise and pitfalls of (the fashionable term) Big Data. He addresses whether promise of capturing exponentially growing volumes of data will obviate need for theory, and even scientific method. The pitfall (or downside) to it is that assembling of massive reams of data might be distracting, and provide little knowledge about truth. Indeed, too much data can lead one into believing in mistruths that are result of identifying faulty causalities among myriad signals! …