# Two-Step Classification with SVD Preprocessing of Distributed Massive Datasets in Apache Spark
## 3 Proposed Method
### 10V Data  
Big data is a rather generic term which may actually be misleading in a number of significant cases. since the extraction of non-trivial knowledge from them is complicated not only by sheer size but also from other factors such as the lack of structure or random patterns of missing values. In order to describe this kind of data, the term 10V data has been coined. The latter summarizes the following top data properties:  
* Volume: The volume of 10V data greatly exceeds the main memory capacity and, therefore, the data have to be moved to secondary memory and possibly across over a distributed system such as NFS, Minio, and HDFS. Thus, new computational strategies about moving the computations and not the data have to be developed. * Velocity: This factor refers to the rate data are generated or refreshed. Data update rate depends on the application and ranges from milliseconds to hours. The larger the time scale, the bigger the need for a sizeable buffer zone. * Variety: 10V collected data can well be structured in various formats, semistructured, or unstructured according to the collection policies. It is not unlikely the same dataset contains graphs, images, sound clips, maps, video, and text as well as raw measurements in binary blobs. * Variability: Large datasets are bound to contain missing values and outliers, both of which need to be addressed with cleansing and anomaly discovery methods as appropriate. The latter are crucial in improving data quality and facilitate building more efficient pipelines. * Volatility: This factor refers to the useful data life span. Before the advent of 10V data it was not uncommon to store everything in data warehouses. Now there is pressure for more selective strategies. Choosing which entries or which transformed attributes for long term storage is a major topic in data science. * Visualization: Humans tend to understand better the \"big picture\" inherent in the processed and refined knowledge because of the way brain operates. Thus, information visualization may well be the key to successfully conveying a message. Although lately the importance of storytelling \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset Size** & **Precision** & **Recall** & **F1 Metric** & **Time** & **Best Layer** \\ \hline \multicolumn{5}{c}{one-step classifier} \\ \hline
5.000 & 0.639 & 0.646 & 0.641 & 0:09:20 & [31, 35, 25, 20, 12] \\ \hline
10.000 & 0.691 & 0.708 & 0.697 & 0:14:27 & [31, 35, 25, 20, 12] \\ \hline
25.000 & 0.720 & 0.719 & 0.719 & 0:32:22 & [31, 40, 30, 20, 12] \\ \hline
50.000 & 0.707 & 0.732 & 0.721 & 1:01:55 & [31, 40, 30, 20, 12] \\ \hline
75.000 & 0.724 & 0.738 & 0.727 & 1:30:39 & [31, 40, 30, 20, 12] \\ \hline
100.000 & 0.755 & 0.776 & 0.759 & 1:55:59 & [31, 35, 25, 20, 12] \\ \hline
200.000 & 0.775 & 0.778 & 0.775 & 3:59:41 & [31, 35, 25, 20, 12] \\ \hline \multicolumn{5}{c}{two-step classifier} \\ \hline
5.000 & 0.631 & 0.646 & 0.637 & 0:09:00 & [24, 25, 20, 12] \\ \hline
10.000 & 0.717 & 0.723 & 0.719 & 0:14:15 & [24, 25, 20, 12] \\ \hline
25.000 & 0.723 & 0.734 & 0.725 & 0:32:41 & [24, 40, 30, 20, 12] \\ \hline
50.000 & 0.784 & 0.788 & 0.787 & 1:02:01 & [24, 40, 30, 20, 12] \\ \hline
75.000 & 0.753 & 0.746 & 0.751 & 1:29:44 & [24, 40, 30, 20, 12] \\ \hline
100.000 & 0.762 & 0.768 & 0.765 & 1:54:06 & [24, 40, 30, 20, 12] \\ \hline
200.000 & 0.764 & 0.776 & 0.772 & 3:59:29 & [24, 25, 20, 12] \\ \hline \hline \end{tabular}
\end{table} 
Table 14: Multilayer perceptron. techniques has gained traction, visualization remains the best and easiest way to describe sizeable amounts of knowledge. * Vulnerability: Each and every computing device nowadays is a potential security threat. The systems of a data processing pipeline are no exception, especially if they collect measurements with the outside world. * Veracity: Creating ad hoc queries and small scale tests easily is an important parameter in data quality. Given the unstructured nature of 10V data, the development of ad hoc queries is paramount as invaluable insight can be gained from them.