# A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment
## 2 Related Literature Review
### Reference Architectures
Additionally, MC-BDP provides a domain-specific case study evaluation using a mixed-methods approach to provide not only the quantitative measurements needed to rigorously assess the contribution from a positivist standpoint, but also a highly valued qualitative appraisal of its impact and potential applications within the target domain. A reference architecture for big data in the cloud focused on cost and the pay-as-you-go model is offered by [45]. This study presents three implementation possibilities for its reference architecture using stream processing technology offered by major commercial cloud providers, namely, AWS, Google Cloud, and Azure. While some similarities can be drawn between this proposal and MC-BDP, such as a focus on big data stream processing on infrastructure commissioned from commercial cloud providers, there are some fundamental differences, such as the preoccupation with vendor lock-in mitigation. Ref. [45]'s implementation proposals are neither portable nor interoperable with components on different clouds. They state that there was a requirement for each service proposed to be compatible with other services within the same provider (e.g., Amazon Kinesis Streams to provide data to Kinesis applications and Google Cloud Pup/Sub for Google Cloud Dataflow). There was, however, no requirement for the services selected to be compatible with services from other clouds, thus reducing the risk of vendor lock-in. Since vendor lock-in mitigation is one of the motivations of this study, the use of managed services is avoided. The reference architecture for big data proposed by [10] is similar to MC-BDP in that it was designed for a specific domain. While MC-BDP focuses on the domain of big data streams for implementers looking to take advantage of the economies of scale brought about by cloud computing, their study's contribution was developed for the domain of national security. Technology agnosticism is a main concern of the reference architecture proposed by the study [10], one which it shares with the current study. The evaluation of the reference architecture was performed using a simple prototype implementation to analyse a data flow of Twitter sentiment data using stream processing and merge it with a news dataflow obtained through the processing of batch data to produce intelligence. Differently from MC-BDP's prototype evaluation, [10] was based on a typical Lambda architecture implementation where stream and batch data were processed separately before the results were merged. There was no controlled variation in the velocity of data ingress as performed for the MC-BDP evaluation, since the authors' objective was to demonstrate the feasibility of the reference architecture, not measure its performance. Case-specific implementations of such reference architecture within the domain of national security was called for as future work to enable a more thorough evaluation of the proposed reference architecture, similar to what was suggested by participants of the MC-BDP case study following its prototype evaluation within the domain of smart buildings. A security reference architecture (SRA) for big data based on the UML notation was proposed by [51]. Their contribution builds upon other industry-standard reference architectures, particularly the NIST Big Data Interoperability Framework [43], by adding a security dimension defined using UML diagrams. Although their proposition differs significantly from MC-BDP, it is included in this review because they highlight the importance of incorporating security aspects into the early design stages of big data implementation [51]. This is the approach supported by this study and integrated into MC-BDP's security layer presented in Section 4. Ref. [47] derived their reference architecture from studying publications related to four real-world big data implementations: Facebook, Twitter, Netflix, and LinkedIn.