Towards scalable execution of scientific workflows with multi-level heterogeneity
Creators: name: Al-Saadi, Aymen
Publisher: name: No Publisher Supplied
Year: 2024

description: Modern scientific and engineering applications increasingly rely on processing large datasets and executing complex computational workflows on diverse resources. For example, biological sciences use computationally intensive sequencing workflows like 10x Genomics single-cell RNA-Seq by offloading computing-intensive tasks to High-Performance Computing (HPC) systems and performing data analysis on cloud clusters. Computational fluid dynamics uses HPC nodes to simulate the injection of jet fluids while rendering the simulations on cloud instances. Such workflows require executing tasks with heterogeneous computing requirements on different computational platforms. Although these workflows enable various scientific approaches and span multiple scientific domains, they share similar requirements, such as: (i) harnessing resources across HPC and cloud environments; (ii) having precise and finite computational requirements; (iii) requiring coordinated execution of tasks on HPC and/or cloud platforms; and (iv) utilizing CPUs, GPUs, or TPUs within the workflow on HPC and/or cloud. Despite these shared requirements, each use case demands varying levels of heterogeneity across resources, tasks, and platforms. However, existing solutions have failed to fully address these requirements, as they are often dispersed, partial, and tailored to specific domains, platforms, and resources. Effectively harnessing heterogeneous HPC and cloud resources present significant challenges, such as, different resource paradigms, large-scale resource management, performance optimization, and productivity. This dissertation aims to enable the efficient management and utilization of heterogeneous HPC and cloud resources to execute scientific iiworkflow applications at a large scale. Specifically, we adapt computing techniques for data-intensive tasks, integrate scalable tools for workflow execution on HPC, and enable automated resource management across HPC and cloud platforms.First, we addressed the resource heterogeneity aspect of the workflows by developing task HPC awareness for image analysis pipelines to efficiently utilize multiple GPU kernels concurrently on HPC resources, resulting in ~50% improvement compared to state-of-the-art approaches. We further investigated the effect of different workflow designs on execution time and resource utilization. This work resulted in improving the processing of ~22,000 satellite and airborne images in ~2 hours with ~90% GPU utilization, compared to 16 hours previously required.Second, we addressed task heterogeneity by designing and implementing a multi-node MessagePassing Interface (MPI) Python function executor and integrated it with the existing workflow management system. This work enabled the execution of two real-world use cases with ~2,000 workflows on two HPC machines, and resulted in ~99% resource utilization, compared to the 95% utilization obtained without the developed capabilities.Third, we addressed platform heterogeneity and resource management complexities by designing and implementing Hydra, a novel HPC and cloud resources manager and broker with workload management capabilities. The results of this work enabled the execution of sea-level projection use cases on both cloud and HPC platforms. We demonstrated that Hydra can manage the execution of 80,000 tasks on a single cloud resource and 10,000 tasks on both HPC and cloud platforms (Azure, AWS, NSF-Jetstream, NSF-Chameleon), with average overheads of 6 seconds and throughput of ~6,000 tasks/s.Lastly, we addressed the issue of cross-cloud resource selection for both commercial and private cloud providers. We proposed the Hydra resource orchestrator as an extension of the Hydra resource broker with two classes of selection algorithms while comparing their performance and selection results. Our work offered two systematic approaches to select cost-effective or high-performance cloud resources based on user constraints across thousands of resources efficiently and effectively