# A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment
## 1 Introduction
### Rationale  
The risk of vendor lock-in is perceived as one of the greatest obstacles to cloud adoption by companies [18; 20; 24; 32]. Consequently, as this study adopts a cloud consumer's perspective, the mitigation of vendor lock-in remains an important motivation throughout its development. The use of open-source technology is associated with a reduction in vendor lock-in risk [33] while simultaneously promoting collaboration and reuse amongst the community [34]. Another way of mitigating vendor lock-in-related risks is by hosting the cloud infrastructure in multiple clouds. From a business perspective, implementers are less vulnerable to unilateral changes in price or SLAs by single providers when adopting a multi-cloud setup. Furthermore, they can achieve more flexibility by combining offers from different providers, or by changing providers for certain services, but not others, if a more advantageous offer is made available [35]. Motivated to achieve the greatest level of flexibility and interoperability of components across clouds while, at the same time, taking advantage of the clouds' economies of scale, our study proposes a reference architecture based on the use of containers on a PaaS model. As containerised applications include all the environment configuration they need to run and can be deployed to any platform equipped with a container engine, they are fully portable across clouds, thus greatly reducing the risk of vendor lock-in and allowing implementers to shift away from the traditional SaaS model usually recommended for SMEs. This study therefore recommends that big data processing frameworks and other components of the proposed reference architecture be deployed as containerised services. From a technical perspective, a strong motivation of this study is to propose a reference architecture that has a discernible value for SMEs and is therefore reliable and fault-tolerant, since high availability and business continuity have been identified as important requirements for these companies [12; 36]. Within a single commercial cloud, fault tolerance for containerised systems can be provided at three different levels: data centre, zone, and region. These are employed by cloud providers not only to improve fault tolerance, but also to facilitate managing the cloud infrastructure in a divide-and-conquer fashion. AWS has introduced the availability zones concept, which consists of isolated data centres hosted in the same region. Since no two availability zones share the same data centre, the resilience of the infrastructure is increased, even for single-region deployments [37]. An infrastructure that spans across multiple availability zones has been recommended by Netflix following the after-effects of AWS outages in the past [38]. However, zone failures involving multiple availability zones have occurred in the past [39]. Likewise, a DNS disruption reported in 2016 affected Azure customers in all regions, effectively rendering the entire cloud unavailable [40]. Our research aims to explore the most resilient design for a cloud reference architecture and has thus identified an infrastructure distributed not only across multiple regions, but also across multiple clouds as a pattern to aim for. The relationship between different levels of fault tolerance and the cost and volume of data transfer from a cloud consumer's perspective is also investigated in this study. Finally, as the current study focuses on the utilisation of commercial clouds for the deployment of containerised distributed applications, another area of research which is yet to fully mature is that of resource consumption estimation for large-scale containerised systems. Currently, most cluster orchestration technologies allocate resources based on a user's initial request [28; 41]. Since users tend to overestimate the resources required by their applications, this approach leads to under-utilisation and, in the context of cloud computing, unnecessary costs [41].