# A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment
## 2 Related Literature Review
### Reference Architectures  
A reference architecture for big data workflow management systems (BDWFMSs) was proposed by [8] for processing data-intensive scientific workflows in the cloud. Although \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Contribution** & **Domain** & \begin{tabular}{c} **Primary** \\ **Quantitative** \\ **Evaluation** \\ \end{tabular} & \begin{tabular}{c} **Data** \\ **Processing** \\ **Focus** \\ \end{tabular} & \
\begin{tabular}{c} **Infrastructure** \\ **Level** \\ \end{tabular} &
\begin{tabular}{c} **Virtualisation** \\ **Qualitative** \\ **Evaluation** \\ \end{tabular} \\ \hline
[8] & Scientific Workflows & Y & Batch & Cloud & Hypervisor & N \\
[43] & Generic & N & Batch/Stream & Cloud/Bare-Metal & Hypervisor & N \\
[44] & Large Corporations & N & Batch & Cloud/Bare-Metal & N/A * & N \\
[9] & Healthcare & N & Batch/Stream & N/A * & N/A * & N \\
[45] & Generic & Y & Batch/Stream & Cloud & Hypervisor & N \\
[46] & Generic & N & Batch & N/A * & N/A * & Y * \\
[10] & National Security & Y & Batch/Stream & Bare-Metal & N/A * & N \\
[47] & Large Corporations & N & Batch/Stream & Cloud/Bare-Metal & Any & N \\
[48] & IoT & Y & Stream & Cloud/Bare-Metal & N/A * & N \\
[49] & Generic & N & N/A * & Cloud & Container & N \\
[50] & Edge Computing & Y & Stream & Bare-Metal & Container & N \\
[51] & Security & N & N/A * & Cloud & Any & N \\
[52] & Generic & Y & Stream & Cloud & Any & N \\ MC-BDP & SMEOD & Y & Stream & Cloud/Bare-Metal & Container & Y \\ \hline \hline \end{tabular}  
* N/A means not applicable or not addressed by the author(s). \({}^{\dagger}\) Qualitative data was analysed quantitatively.  
\end{table} 
Table 1: Summary of related work and comparison with MC-BDP reference architecture. developed for processing big data using a distributed cloud-based infrastructure, BDWFMSs differs fundamentally from MC-BDP. Firstly, they were developed for the domain of scientific workflows, whereas MC-BDP is targeted at SMEODs entering the realm of big data stream analytics. Consequently, BDWFMSs do not specifically promote a multi-cloud environment, nor are they motivated by the desire to mitigate the risk of vendor lock-in. Virtualisation is provided at virtual machine level for BDWFMSs while MC-BDP uses container-level virtualisation. Thus, BDWFMSs are more significantly affected by the lack of standardisation in virtual machine models and specifications offered by each provider than MC-BDP. When it comes to the experimental evaluation of BDWFMSs, their design shares some similarities with MC-BDP's, such as the introduction of a computationally intensive calculation into some of the processing functions to observe how the system performs at higher loads. Both the BDWFMS and MC-BDP used real historical data for their experimental setups: the former looked for patterns in driving data for fifty New York drivers over the course of an hour for its first case study and analysed astronomical images for its second case study [8], while the latter calculated the energy efficiency of a data centre based on energy consumption records collected over the course of a year as part of its case study. An important difference between the experimental setups utilised to evaluate both reference architectures was that MC-BDP used a simulator to emit the data in real-time, whereas the BDWFMS analysed static data in its case studies. This difference can be explained by MC-BDP's focus on big data stream processing, whereas the BDWFMS's main concern is the processing of batch data. The big data reference architecture was developed by the National Institute of Standards and Technology (NIST) [43]. NBDRA is composed of five different roles (system orchestrator, data provider, big data application provider, big data framework provider, and data consumer) and two fabrics (security and management), each implemented independently based on functional requirements.