# A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment
## 1 Introduction  
### Research Context  
The advent of integrated yet pervasive systems such as the Internet of Things (IoT), the Internet of Everything (IoE), Fog/Edge computing, and Cloud computing has brought big data to the forefront of technology research. Smartphones, tablets, GPS trackers, sensors, and video surveillance devices produce vast amounts of data in varying formats in real time. Consequently, this poses challenges not only in terms of storage, but also in terms of timely processing and analysis for intelligence. Big data-related challenges are related to high volume, velocity, veracity, and variety. Historically, systems which focused on the volume aspect of big data appeared first. Those are known as batch architectures, as static finite data is stored in files and processed in batches. The most prominent and most highly utilised of such systems is Hadoop, an open-source distributed processing system based on the MapReduce algorithm developed by Google [1]. Hadoop is still currently popular but has evolved to include a complete ecosystem of open-source applications for batch data processing. However, the main critique of Hadoop and other batch systems is that they overlook two important aspects of big data: velocity and the fact that the data source is potentially infinite [2]. Batch systems were not designed to process streaming data and produce results in real time or close-to-real time. Following the first generation of big data processing systems, which focused on the processing of batch data, stream systemswere developed to process a potentially infinite source of data arriving at high velocities in close-to-real time. The focus of these systems was different; therefore, a new architectural design and approach to data processing was necessary. The concept of a window was introduced, which involved breaking up the data streams into manageable finite quanta for processing. This was facilitated by applying two functions: (1) assign a timestamp to arriving data; (2) based on the timestamp, assign data to a time window for processing. Thus, time windows are abstracted groups (with a finite number of records) used for processing [3]. In order to address late and out-of-order streaming data, watermarks were introduced to represent the time by which all records for a given window are expected to have arrived [4]. They are used, in conjunction with a defined tolerance, to signal that a window is ready for processing. The Lambda architecture challenged the two main big data paradigms, namely, the batch and stream models, to propose a model where both technology stacks worked synergistically. Both batch and stream stacks were to be separately maintained, and data processing would be independently undertaken, with the results subsequently merged to obtain a combined view of the data. Although this architecture has been critiqued due to the need to create and maintain complex processing code in two separate places, thus incurring additional operational managing costs for two different technology stacks [5], it is still deployed in real-world applications (e.g., Facebook [6] and Twitter [7]). Though our proposed reference architecture does not completely address the Lambda architecture overhead-related problem, it exploits the use of a super-framework to enable the same processing code to be run on batch or stream infrastructures, thus resolving the code duplication issue. Additionally, it advocates multi-tenancy so that both batch and stream frameworks share the same cluster. Currently, there are existing hybrid architectures where batch-based architectures have been adapted to process streaming data, as well as vice-versa, where stream-based ones have been adapted to process data from batch files. Although, currently, a plethora of big data architectures is available, there is a discernible need for the development of reference architectures, defined in this study as template architectures for a given domain.