# A Comparative Analysis of Big Data Frameworks:
## 5 Comparative Analysis of Big Data Frameworks
### Feature Vector Related to Support for Machine Learning  
Machine learning is a powerful tool that enables the use of data to make predictions and help to make decisions. The core of machine learning is data that empowers the underlying models. The emerging technologies of big data processing heavily incorporate machine learning tools to help make smarter and more informed decisions based on data.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **Hadoop** & **Spark** & **Storm** & **Samza** & **Flink** \\ \hline \multirow{2}{*}{Latency} & high (seconds) & low (few seconds) & very low (sub seconds) & very low (sub seconds) & very low (sub seconds) \\ \multirow{2}{*}{Throughput} & high & High & medium & high & high \\ \multirow{2}{*}{Scalability} & high & Moderate & moderate & low & high \\ \hline \hline \end{tabular}
\end{table}
Table 6: Comparison of big data frameworks with respect to performance related features.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **Hadoop** & **Spark** & **Storm** & **Samza** & **Flink** \\ \hline \multirow{4}{*}{Fault detection} & heart beat & heart beat & \multirow{4}{*}{heart beat} & \multirow{4}{*}{\begin{tabular}{c} Keeping in-memory \\ record of all emitted \\ tuples and tracks \\ them within a \\ configured timeout \\ \end{tabular} } & \multirow{4}{*}{\begin{tabular}{c} \\ heart beat mechanism \\ \end{tabular} } \\  & mechanism & mechanism & mechanism & \multirow{4}{*}{\begin{tabular}{c} Keeping in-memory \\ record of all emitted \\ tuples and tracks \\ them within a \\ configured timeout \\ \end{tabular} } & \multirow{4}{*}{\begin{tabular}{c} \\ heart beat mechanism \\ \end{tabular} } \\  & & & & \\ \multirow{2}{*}{Fault Recovery} & Data Replication & RDD lineage & \multirow{2}{*}{\begin{tabular}{c} Tuples acknowledgment, \\ Zookeeper \\ \end{tabular} } & \multirow{2}{*}{Change log capturing} & \multirow{2}{*}{\begin{tabular}{c} Light weight \\ distributed snapshots \\ \end{tabular} } \\  & & & & & \\ \multirow{2}{*}{\begin{tabular}{c} High \\ availability \\ \end{tabular} } & \multirow{2}{*}{\
\begin{tabular}{c} Redundant standby \\ NameNodes and \\ ResourceManagers \\ \end{tabular} } & Multiple & Running & Relies on YARN’s & \multirow{2}{*}{
\begin{tabular}{c} Stores JobGraph and \\ all necessary metadata \\ on a reliable remote \\ storage system \\ \end{tabular} } \\  & & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 7: Comparison of big data frameworks with respect to features related to fault tolerance.  
Since machine learning concepts and algorithms are increasingly used in big data analytics, thus almost all of the frameworks are complemented with machine learning libraries and toolkits. Though, no special platform or library is required to execute machine learning tasks on Hadoop clusters, yet, a set of machine learning packages that can be run on Hadoop includes Mahout, H2O, Distributed Weka and Oryx (2010). Spark and Flink have their native machine learning libraries. Spark's MLlib aims to simplify machine learning tasks for Spark. Its core functions include clustering, classification, collaborative filtering and regression. Additionally, it also includes algorithms for dimension reduction, transformation, optimization and feature extraction ([PERSON] et al., 2015). FlinkML is Flink's native machine learning library which was released in April 2015. It aims at providing extensible machine learning tools and algorithms for the development of complex machine learning applications. Storm and Samza both do not offer native machine learning library. However, both have compatibility with SAMOA ([PERSON] and [PERSON], 2018). As can be seen from Table 8, that all five frameworks have support for either native or in compatibility mode for machine learning tools that implements clustering, classification and regression algorithms. However, H2O is the only machine learning platform considered in this paper that implements deep learning algorithms, consequently, those frameworks that can be integrated with H2O offers deep learning algorithms.