# Occam's Razor for _Big Data_? On Detecting Quality in Large Unstructured Datasets
## 3 Big Data Analytics: From Machine Learning to Artificial Intelligence  
A major challenge for _big data_ analytics consists of the accurate and fast segmentation of large datasets consisting of parameter values, generally expressed in numbers, into quality clusters, i.e., those which optimally define the input, which is less than a straightforward matter. For example, for a dataset with \(n=1000\) feature rows, there would be \(10^{301}\) possible combinations to evaluate in cases where prior knowledge relative to how the data are organized is unavailable. While the current trend is to believe that artificial intelligence and deep learning in increasingly larger neural network architectures will provide the all-encompassing answer to questions on how to address this problem, the scientific principle of parsimony (Occam's razor) implies starting by trying to improve the accuracy of data mining without \"reinventing the wheel\" altogether. This should be possible by building on machine learning algorithmic approaches which have proven their worth. When combined adequately in a building block approach, simpler approaches may prove far more powerful than other, more complex algorithms and improve the quality of data mining and software analysis, as will be illustrated later herein on the basis of specific examples. Machine-learning (ML) systems [21] identify objects in images and select relevant results of search. The classic supervised machine learning approaches were initially limited in their ability to process data in its raw form and the analytic systems required specific domain expertise to program feature extractors that transform raw data into a valid internal representation vector for learning and input classification. This \"old-fashioned\" way of programming had the considerable advantage that the result obtained could be linked to well-identified and controlled steps in the machine-learning procedure. In the current context of _big data_ analytics, unsupervised ML algorithms prevail. We propose that the most prevailing analytic approaches to _big data_ may be arbitrarily ranked into three categories: (1) _'Educated'_, (2) _'Wild'_, and (3) _'Artificial Intelligence'_ (AI); an overview is given in Figure 2.  
Unsupervised ML methods of the 'educated' type (Figure 2, top) imply educated guesses (hypotheses) relative to the data structure and are aimed at generating predictions. The most common of these would be hidden Markov models, which have been applied in the _big data_ context to generate spatio-temporal predictions based on clusters of interest in large video datasets [22], and association analysis or rule-mining, aimed at finding frequent items in a database with the least complexities to generate predictions. Association analysis for _big data_ is limited by the problem of finding large enough  
Figure 2: Analytic approaches to _big data_: From unsupervised machine learning (‘educated’ or ‘wild’) to artificial intelligence (AI). item clusters in datasets [23]. Making assumptions about trends in large data to generate predictions makes no sense if nothing about the quality of the data is known a priori. The unsupervised ML methods of the 'wild' type (Figure 2, middle) make no particular assumptions about the data, and are therefore a first choice for detecting quality and structure in unknown _big datasets_. These different methods can easily be adapted and combined for the identification of meaningful groups in large and unknown datasets. They employ algorithms for data clustering [24; 25] and dimensionality reduction, often by principal component analysis [24], to reduce the size of very large data in high dimensional (generally Euclidean) space to smaller sets of weighted data points, or core datasets. Essential properties of clustering algorithms as a first-choice analytic solution for detecting quality in _big data_ will be pinpointed further in a subsection here below.