# A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment
## 4 MC-BDP Prototype Evaluation  
The ever-increasing volume, velocity, and variety of big data means that architectures designed to process it need an infrastructure capable of expanding accordingly. Moreover, since big data processing is generally performed by distributed nodes, there must be provisions in place to ensure that the system continues to operate normally should one of the nodes fail. Both these qualities are most adequately provisioned by cloud computing, \begin{table}
**Technology** & **Purpose** & **Alternatives Considered** & **Rationale** \\ \hline Virtual Machines from & Run Kafka server, orchestration, data processing, and monitoring containers & Virtual machines from different providers, private cloud, bare metal, CaaS & The experiments were designed to evaluate a multi-cloud setup using commercial providers. Grants were received from the providers selected. \\ \hline Linux Operating System & Run containers on a PaaS model & Windows, macOS, ChromeOS & Native integration with Docker, open-source, and free of costs. \\ \hline Docker Containers & Run the big data framework’s data processing containers & Linux Containers (LXC), OpenVZ, or Linux-VServer & Most popular container technology, open-source. \\ \hline Docker Swarms & Container orchestration & Kubernetes, Mesos & Although Kubernetes is more feature-rich and more widely used, Docker swarms are simpler to configure and native to Docker. \\ \hline Weave Net & Container networking across clouds & Pipework, Flannel, Open vSwitch & Simple to install, support for multi-cloud networking. \\ \hline Kafka & Stream data source. & Pub/Sub, Kinesis, RabbitMQ & Open-source, persistent, scalable, suitable for high data throughput. \\ \hline \hline \end{tabular} \
\begin{tabular}{c c c} \hline \hline  & &
\begin{tabular}{c} Flink was, at the time of \\ implementation, the most capable \\ open-source runner for Apache \\ Beam [95]. It was also available as a set of containerised services for Docker. \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Summary of technologies used in MC-BDP prototype implementation. which offers scalability and fault tolerance at potentially lower costs due to its economies of scale. However, an innovative approach to utilising commercial clouds from the perspective of an implementer wishing to mitigate the risk of vendor lock-in is needed, and MC-BDP was created to fulfil this demand. The experiments in this section measure container CPU and memory utilisation, as well as network utilisation in clusters of different sizes, to understand the impact of scaling up or down on resource consumption. Likewise, the same metrics are used to understand how different configurations of containers deployed to different clouds may have an impact on resource consumption in a scenario of fault tolerance, e.g., when one of the nodes becomes unavailable mid-processing. Being able to write the data processing code once and run it in different platforms and frameworks has also been identified as a desirable requirement for companies who may not have the technical expertise in-house to maintain complex bespoke programs. Moreover, preventing duplication and promoting reuse is an accepted software development principle which implementers should strive to observe in order to promote efficiency and enhance the maintainability of the system in the long run [96]. Technology agnosticism is enabled, in this prototype implementation, by using the Beam framework. The experiments in this section aim to compare CPU, memory, and network utilisation in clusters running equivalent Beam and Flink processing code to determine whether one, which is technologically agnostic, has a significant advantage over the other, which is not. Finally, an understanding of how different windowing rates and container co-location configurations affect resource utilisation was needed so implementers could make more informed business and architectural decisions, particularly in the cloud, where the financial impact of such decisions is more readily felt.