Is Big Data Overrated? The underestimated innovation challenges of BD management

The exponential growth of data all over the world in the last 15 years (McAfee & Brynjolfsson, 2013), has originated the field of studies of big data (BD). The literature on BD has studied the potential benefits associated to the analysis of large quantities of data and how this can be used to improve the organization; from its operations to the development of new products (for example: Gandomi & Haider, 2015; McAfee & Brynjolfsson, 2013). Furthermore, data has been portrayed as the key to solve some of the current societal problems (Patterson, 2017). Additionally, the current big disruptions of society and business have a digital nature, as for example the creation and use of artificial intelligence in organizations. However, for this data to be translated into value, it must be first transformed into knowledge (Zins, 2007). To this end, it is important to take into account that this data production is directly connected to the humans, as it is mainly produced by them or about them (McAfee & Brynjolfsson, 2013). Therefore, ignoring the human component of the current level of data may only result on missing out on its potential. There is an important effort to be made as to consider the human factor in the analysis of data being currently produced since it has the potential to be used as to develop new products and services that may contribute to solve some of these important social problems. To this end, we present a new experimental method based on BD analysis that integrates the human perception for new product development (NPD). This way, taking to a higher degree advantage of the data potential. This method is called Human-in-the-loop (HITL), a model with human interaction used for modeling and simulation for the cinematographic and gaming industry (Rothrock & Narayanan, 2011). Here is adapted as an experimental method integrating human perception into the analysis of BD. It is composed by two dynamic loops of data analysis and a method of connection. The first loop regards the human interaction through the introduction of human behavior in the form of experiments or ethnographic data. The second loop regards computer-based analysis of structured data impulse introducing new data sets. The mixed method of digital anthropology serves as the bridge to link structured and unstructured data types. However, as the case shows, there are risks to the implementation of BD in the organization; as producing better answers for organizational problems brings inevitably new questions that require unforeseen resources (George, Osinga, Lavie, & Scott, 2016). Through the take-aways from this case study, as well as the literature on the topic, we propose a theoretical contribution using the framework of George et al. (2016). We establish that given the current technology of data analysis there are challenges for data-driven innovations depending on the data scope and granularity that each case regards. More specifically, we state that the current BD technology is unable to confront simultaneously high granularity (variety) and scope (volume) for analysis. Therefore, when companies try to give answer to the new questions that BD brings along, they enter the unknown, represented by what we define as the ? gravitational sink’ of BD. This is defined as such, as much of the solutions explored by organizations will fail into this sink. However, due to the potential of disruption of the remaining technologies promised by the technology-driven literature (Habtay, 2012), organizations are attracted to its exploration. Additionally, we argue that ambidextrous organizations are in a better position to undergo the challenges of the implementation of BD technologies as exploration and exploitation should happen as a dualism due to the dynamic nature of the BD technologies. Therefore, the research objective of this paper can be summarized in the following question: What is the role of data granularity and scope in the management of digital disruption and innovation for BD? Finally, we provide a guide for practitioners, showing which standardized technologies available are the best fit for each data scenario. This way allowing for certain level of exploration; yet avoiding the gravitational sink. For example, when the scope of the data regards a small volume with no variety, it can be handled with statistics, or manual data processing. Once the data scope increases, BD processing can help handling large volume of data. Tools for business intelligence can effectively analyze this volume in absence of variety. Additionally, when the variety increases, tools as NLP or image recognition provide a solution. Once these technologies are in play, the resources allocated for data analysis grow due to the problems presented by Del Vecchio et al. (2018): need of skilled analysts, increasing data storage and processing cost, security risks, etc. When the data presents increased variety and a small volume, traditional ethnographic methods handle data thickness in the unstructured data (Wang 2016). Nevertheless, once the data volume increases, digital ethnography is required to analyze larger data sets. These methods, help explore new questions, but none of them fully answer high volume and variety data related problems. This exploration of the unknown represented by the gravitational sink presents high level of challenges due to its dynamism and continue need of exploration. However, it also has the potential for long lasting disruption. George, G. et al. (2016) 'Big Data and Data Science Methods for Management Research', Academy of Management Journal, 59(5), pp. 1493-1507. Habtay, S. R. (2012) 'A Firm-Level Analysis on the Relative Difference between Technology-Driven and Market-Driven Disruptive Business Model Innovations', Creativity and Innovation Management, 21(3), pp. 290-303. Jansen, J. J. P., Van Den Bosch, F. A. J. and Volberda, H. W. (2006) 'Exploratory Innovation, Exploitative Innovation, and Performance: Effects of Organizational Antecedents and Environmental Moderators', Management Science, 52(11), pp. 1661-1674. Del Vecchio, P. et al. (2018) 'Big data for open innovation in SMEs and large corporations: Trends, opportunities, and challenges', Creativity and Innovation Management, 27(1), pp. 6-22. Zins, C. 2007. Conceptual approaches for defining data, information, and knowledge. Journal of the American Society for Information Science and Technology, 58(4): 479-493.