Multi-stage cloud framework based on agents for dynamic, scalable, and secure distributed computing
Authors:   , Gargees Rasha S. 
Published: August 2021

[ACCESS RESTRICTED TO THE UNIVERSITY OF MISSOURI-COLUMBIA AT REQUEST OF AUTHOR.] We have reached the point of ubiquitous sensing as we continue to witness the explosive growth of the Internet of Things (IoT) and other widespread sensor platforms. The data deluge that results from the overwhelming capacity of these sensors represents a continuing challenge for data processing in support of data analytics. The challenge is compounded by the diversity and variety in this data, as it may be derived from several sources, including various sensing modalities, human activity trace data, or media data. The very heterogeneous nature of this data necessitates the development of modular and extensible data processing pipelines to accommodate these diverse sources of data and the litany of potential analytical use-cases. Herein lies another challenge, namely accommodating the ever-increasing need for flexible and scalable computing capacity that can operate within or across different computing localities, such as local computing clusters or cloud computing. In each computing environment, some considerations are necessary regarding scalable compute power, storage, and specialized data processing. For this reason, it becomes vital to develop a flexible framework that can efficiently manage the complex data, the complex computation pipelines, and specialized computing needs. In this dissertation, a generalized distributed computing framework has been developed for both batches and streams of big data workloads. The framework enables multi-stage processing that leverages computational distributed agents to handle large-scale data in parallel to deliver high-throughput computing (HTC). The computational agents are distributed both in breadth (parallelism) and depth (various stages) to enable the scalability, parallelism, and heterogeneity of modern computational big data work flows. Additionally, the proposed framework allows the interleaving of computation among the stages to speed up the processing and support computationally intensive applications, which further enables it to leverage the resources to the highest possible extent, such as high-performance computing (HPC) elements. Three connection topologies using the publish-subscribe pattern for the intelligent computational agents are investigated between the stages, which include: fully connected, partition, and mesh. The framework is dynamically extensible and supports analytics from the ground up by incorporating localized (within compute nodes) and centralized persistent data storage (PDS). Two use-cases were evaluated to demonstrate the capacity to handle diverse workloads and support a variety of analytics, covering both stream and batch workloads. First, a visual object detection from a live data stream of IoT cameras is evaluated on local virtual machines and cloud infrastructure. Second, a satellite imagery batch data processing pipeline is instantiated that leverages deep neural networks and unsupervised machine learning for visual content cluster analysis in large geospatial imagery datasets. We show that the framework utilizes the CPU, memory, and network resources efficiently; and demonstrate that the framework scales in both breadth and depth of data processing pipelines using parallel computing nodes and achieves excellent data goodput. We evaluate a variable number of agents in each stage, as well as the three connection topologies. We have measured the data velocity and compute agents active state (CPU, RAM, and internal data structures) to understand the system behavior in both the batch and stream workload scenarios, as well as the bursting data streams. Ultimately, a security layer has been developed to protect sensitive data. Three schemes of data encryption have been investigated based on the status of the data; data in motion encryption, data at rest encryption, and end-to-end encryption. The performance evaluation of the system has shown that the proposed schemes have maintained the lightness of the framework while ensuring security and authorization.