# On a Certain Research Gap in Big Data Mining for Customer Insights
## 4 Challenges in Big Data Mining for Customer Insights
With the prodigious size and variety characterizing big data, such centralized approach may not be used. There is a strong need of more distributed approaches capable of mining huge amounts of unstructured data [159]. Some other challenges include, e.g., lack of large-scale data representation (for mining purposes), lack of effective and efficient online large-scale machine learning techniques, and lack of data confidentiality mechanism [77]. Challenges concern mining algorithms which must deal with sparse, uncertain, incomplete, complex, and dynamic data [92]. Moreover, the constant inflow of data to be mined can be recognized a momentous challenge, as many mining algorithms do not provide proper sequences or patterns [143]. Some of the proposals to overcome this obstacle include, e.g., incremental pattern mining and cluster analysis, when the discovered patterns and clusters are incrementally augmented with updated information [160], post-processing enhancements of mined patterns [138], and special spatio-temporal representations of data for further mining [77]. Therefore, these stages, such as data cleaning, integration, ranking, and querying, are often considered as the sources of \"algorithmic bias\" [161; 162]. Reasoning about them as well as attenuating inequity upstream from the final data analysis phase is potentially more impactful [162]. However, it becomes obvious that for valuable insights from big data mining it is essential to consider temporal-related issues. As [PERSON] et al. [92] point out, in a dynamic world, data and information representing interesting features from the environment of an enterprise enlargement. Hence, while mining useful patterns from big data, it is indispensable to consider these evolving changes. However, it seems symptomatic that a miniature number of big data analytics definitions even mention the question of dynamics. Figure 4: Conceptual classification of big data challenges. Source: based on [PERSON] et al. [151]. For example, [PERSON] et al. [5] while presenting sample definitions of BDA consider only two ones addressing the dynamic dimension of BDA. A challenge in big data mining hence arises--how to deal with dynamic/temporal aspects of the realm described by big data. One of the ways to do it is to implement agile big data analytics. BDA is seen as a 'bridging' instrument in development of software applications using agile methods [79]. Agility is achieved by creating a data infrastructure enabling identification and evaluation of various big data sources [11]. Afterwards, there are approaches focused on big data stream processing enabling a flexible mining solution [159]. However, these solutions are insufficient when it comes to real-time data processing [117]. The real-time big data analytics presents another challenge related to big data mining which must be considered [76]. Many of the phenomena of interest to the organization are represented as time series [163], this applies to, e.g., sentiment analysis or user's website activities. However, many other phenomena are too intricate to be represented this way. Knowledge coming from organization's environment evolves very quickly because of a constant inflow of data and information. The big data mining in real time may ameliorate decision-making processes in organizations because it would enable dealing with real-time uncertainties [88; 120]. The time dimension of big data is reflected in the speed of their inflow. This causes big data to be transient which implies the need to mine them as and when they are generated [139]. The timeliness of data analysis and mining is the succeeding challenge, tightly linked with the challenge of dealing with temporal dimension of big data. This timeliness challenge is discussed in [158] in more detail. The most intuitive way to deal with temporal aspects of big data mining is to treat the data inflow as a set of events.