# A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment
## 3 MC-BDP Reference Architecture
### MC-BDP Prototype Implementation  
This section summarises the case study based on which a prototype was created to evaluate the MC-BDP reference architecture. It also describes the results of MC-BDP's quantitative evaluation. A needs analysis exercise conducted at the start of our research revealed the existence of previous research conducted in 2012 with the Sustainability Centre at Leeds Beckett University to assess the energy efficiency of a data centre by calculating its power usage effectiveness (PUE) [58], and our current study leveraged more advanced technology with an aim of providing close-to-real-time data processing. Figure 3 summarises the original PUE calculation process: energy consumption readings from IT equipment and from the rooms in the data centre building were sent to an EGX300 integrated gateway server. The data was then extracted into Excel reports as an ad hoc process. Notably, the readings were emitted every 30 min, and the log extraction into Excel took 43 min to run [58]. Moreover, the reporting was produced retrospectively on a pull basis. A simulator was developed to emulate the behaviour of the EGX300 integrated gateway server at the Woodhouse Data Centre at Leeds Beckett University. The components involved in the prototype implementation of MC-BDP and their interfaces are shown in Figure 4. The energy consumption simulator provides a simple web-based interface through which a user can launch a simulation (e.g., transmit energy consumption records every 5 s for 10 min). It interfaces with the energy consumption producer component which receives historical readings from a database hosted in the Azure cloud, uses the data interpolator to generate realistic readings based on the historical readings, and emits the results to a Kafka server in the frequency requested. The PUE processor component is a distributed big data stream processing pipeline which runs on the Flink framework. It consumes the energy readings from Kafka, calculates the PUE, then exposes the results which, in the example depicted in the diagram, are posted to an Azure Event Hub. Different configurations were used in the development, including logging the results to a file to verify accuracy and posting them back to a different Kafka topic. Microsoft Power BI integrates easily with the Azure Event Hub and was used to create a simple dashboard to display the PUE calculations in real time. The formatted PUE results from the Azure Event Hub were also stored using Azure Blob Storage as proof of concept. The framework and orchestrator components shown on the bottom right-hand side of the diagram show how the processing job is launched and the parallelism is set through the job manager interface. The work is then distributed across several parallel task managers. The number of job managers and task managers available is controlled by the orchestrator, since these components run as containerised services, and the interface to scale these up or down is also exposed. Figure 3: Original PUE calculation process at Woodhouse Data Centre. The EGX300 integrated gateway server simulator is used to transmit data for the prototype implementation. The simulator comprises three modules: a data interpolator, a transmitter to transmit readings at desired frequencies, and a front-end user interface. The data interpolator component is a simple Maven project with a single class and a single static method. The static method takes an original consumption record for a given duration as a parameter and outputs simulated consumption records for a shorter duration contained in the initial duration. Instead of using a simple average to calculate how much energy was consumed for smaller durations contained within the original duration, the consumption values generated vary randomly, up to a maximum variation percentage passed in as a parameter.