# A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment
## 3 MC-BDP Reference Architecture  
### Methods  
A gap was initially identified in the literature, reflecting a shortage of systematic academic studies where container technology was applied to the domains of big data processing and cloud computing [21]. Additionally, reference architectures with a focus on cloud consumers based on models other than SaaS are under-studied. To address this gap, our research aims to leverage the latest advances in containers and container orchestration technology for a new PaaS-based multi-cloud reference architecture targeted at big data stream processing.  
As part of the requirements engineering phase, we conducted a critical review of published work by three major big data corporations: Facebook, Twitter, and Netflix [31; 57]. Ten non-functional requirements were identified and discussed in the context of these companies' architectures: batch data, stream data, late and out-of-order data, processing guarantees, integration and extensibility, distribution and scalability, cloud support and elasticity, fault tolerance, flow control, and flexibility and technology agnosticism. Based on these requirements and industry's best practices gathered from real-world implementations, a new MC-BDP reference architecture for big data stream processing in a multi-cloud environment was designed and developed. Subsequently, this architecture was implemented as a prototype in an energy efficiency case study involving the Estates and Sustainability departments at Leeds Beckett University. This case study used real data obtained by a previous data centre energy efficiency study [58] to calculate its PUE (power usage effectiveness). Since the mechanisms in place to emit consumption data and calculate the PUE have not evolved since the original study took place, the data collected remained relevant. The case study targeted by our research has been selected because it has characteristics which are appropriate for the testing of our prototype [59]. A needs analysis interview revealed that the Estates and Sustainability departments at Leeds Beckett University managed data which had inherent big data characteristics: volume (log files with operational data that spans several years), velocity (data from meters that is sampled and streamed in real-time), and variety (device log exports, for example, contain unstructured data, as the format is specific to each manufacturer). The real-time sampling rate used for the data centre energy efficiency study was, however, limited by the technology available at the time. For the purpose of this research, the data granularity has been enhanced to meet the higher volume and velocity requirements together with MC-BDP's inherent strategy for dealing with late data [60]. Due to operational and availability constraints, a simulation exercise was carried out for the evaluation. Instead of using the real EGX300 server located at one of Leeds Beckett University's data centres, a simulator was written to emulate the behaviour of this server. The simulator used real energy consumption data obtained from the previous power usage effectiveness study [58] and readings were transmitted at desired frequencies using an interpolation algorithm. While using a simulator carries lower risks and is time and resource-efficient, it has the disadvantage of not providing real-time insight into live data. The main purpose of this case study was to conduct a technical evaluation of the proposed reference architecture. Three out of the ten non-functional requirements were selected: scalability, fault tolerance, and technology agnosticism. This selection is supported by current research in the field, as scalability and fault tolerance have been identified as key issues in a systematic literature review of big data stream analytics [61]. Correspondingly, a major factor of concern, the vendor lock-in risk, was mitigated through technology agnosticism. The seven remaining functional requirements were not evaluated due to the time and scope limitations of this project.