# A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment
## 5 Conclusions and Future Work
### Limitations of the Study
#### 5.2.1 Internal Validity  
The main limitations to the internal validity of MC-BDP's experimental evaluation were the measuring frequency, the checkpointing frequency, and the monitoring metrics utilised. Usage statistics were gathered from the host and sent to the Azure Monitoringservice. These measurements were subsequently queried using the Azure Log Analytics service. A collection sample interval of ten seconds was used. The rationale for keeping this configuration was that it constituted the greatest granularity available to describe a time interval of five minutes (thirty readings). At the time the experiments were designed and conducted, the minimal collection sample interval allowed for Linux performance counters was ten seconds [111]. It is believed that, should this technical limitation be lifted in the future, it will be worth experimenting with higher sample frequencies, provided that they do not significantly interfere with the performance of the host machine. Alternatively, further studies could be designed using other monitoring solutions such as Prometheus [112] to better understand the effect of different data collection strategies and technologies on the metrics observed. The checkpointing frequency used by the fault tolerance experiments is another configuration value which was kept constant in the run of the experiments conducted. Checkpointing was configured to honour 'exactly-once' processing guarantees, take one snapshot every five hundred milliseconds, and to use the internal memory of the machine where the job manager was running for storage. The timeout for completion of the snapshot operation was set to one minute. There was no limit to the number of execution retries, but the retry delay was set to five hundred milliseconds. The rationale for choosing those values was to achieve as close as possible a result to exactly-once processing while, at the same time, ensuring that the overhead introduced by taking the snapshots did not significantly interfere with data processing. This had to be true even for the scenario where a node was suddenly lost during the highest volumes/velocities of data ingress and processing. The percentage of data loss was monitored during the fault tolerance experiments. It was also used to fine-tune the checkpointing configuration in test runs of the experiment. Once an acceptable level of data loss was achieved (less than 2%), the configuration was saved and treated as a constant. This study believes that the checkpointing frequency is an experimental setting which could be explored further. A thorough understanding of the relationship between checkpointing frequency, resource utilisation, and percentage of data loss could be particularly useful to architects and implementers of big data stream systems, as could the inclusion of their associated costs when utilising commercial cloud providers. In terms of the monitoring metrics utilised, one of the limitations of using the Azure Container Monitoring Solution is that it only gathers data at the container level. If node and cluster-level data is needed for further comparisons, either a different solution must be used or the container level data must be aggregated, leaving it susceptible to approximation errors. The adjusted queries detailed in Figure 11 were designed with the help of the Microsoft Log Analytics Team to enable this research to compare the resource utilisation of containers deployed to virtual machines of different specifications. The solution encountered, which aggregates the container utilisation measurements by virtual machine using different functions, has its limitations. The approach is coarse-grained, particularly with regards to calculating maximum utilisations at the node level by employing a technique not dissimilar to windowing in order to assimilate out-of-order, late, and missing data. Suggestions for improvement involve implementing node and cluster-level monitoring using technologies other than the Azure Container Monitoring Solution.