# A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment
## 3 MC-BDP Reference Architecture
### Experimental Procedure  
The experimental procedure common to all simulations is summarised in Figure 1. It starts with booting all participating machines, which are connected to the internet via SSH secured by public key authentication. The decision to use a public key instead of password authentication was informed by security considerations at the design stage of the prototype, as recommended by the Cloud Computing Adoption Framework (CCAF) proposed by [62]. Public key authentication is more secure than password authentication given that, in the event of the server being compromised, password authentication confirms a valid username/password combination to a potential attacker [63]. The next step is to run a command in each machine to register with the swarm manager as a worker. As part of this step, the machine's IP address is sent to the swarm manager via SSH. Since the Azure machines have static IP addresses, this step must be completed only once for all experiments. The Google Cloud machines are configured with ephemeral IP addresses, as static IP addresses are not available for purchase. For this reason, they must be registered with the swarm manager after each reboot. The Flink container services were launched from the Azure swarm manager virtual machine via the Docker swarm interface, accessible through the SSH console. The Flink job manager service used the multi-cloud network created using the Weave plugin and was constrained to run on a dedicated machine. This constraint was introduced to isolate the  
Figure 1: Experimental procedure for all simulations. job manager service from the task manager service running on worker nodes, thus allowing for more granular monitoring and gathering of performance statistics. The image used to create the job manager service was obtained from the Docker Hub repository and was minimally modified. The Flink task manager service, distributed across different containers, also used the network created using the Weave plugin. A constraint was added to run this service on any machine but the one dedicated to run the job manager service. Once the task manager service was up and running, it was scaled to the desired level of parallelism using the Docker swarm interface. Once the data processing job is running, initial statistics are gathered using the Weave Cloud service [64]. CPU and memory utilisation metrics are verified to check for discrepancies such as readings which are abnormally high or low. Any reading above 2% or below 0.1% for CPU utilisation was considered abnormal, and the setup process was restarted with a fresh reboot of the machine where the abnormality was observed. Similarly, any reading above 600 MB or below 200 MB for memory utilisation was considered abnormal and warranted a reboot of the machine and a restart of the setup process. With the data processing job running and ready to receive data, the energy consumption simulator job is started using the graphical user interface. It runs for five minutes, transmitting data at the desired frequency. Once the energy consumption simulator finishes transmitting data, it displays the start and end transmission times. These are noted and used to set the times that the Azure Log Analytics queries are run 'from and to'. The setup is then cleared to prepare it for the next experiment: the PUE processor job is stopped via the Flink job manager interface, the Flink job manager and task manager services are stopped using the Docker swarm interface, the nodes are removed from the swarm, and the virtual machines are switched off. The final step in the process is to run the Azure Log Analytics queries using the Azure Portal. The 'from and to' times are set to the start and end transmission times. The queries executed are shown in Figure A1.