Data-driven triggers for optimizing staging-based in situ scientific workflow
Creators: name: Wang, Zhe
Publisher: name: No Publisher Supplied
Year: 2022

description: In situ processing alleviates impacts due to the gap between computation and I/O capabilities of current large-scale computing systems by running the simulation and data processing components of the workflow on the same system and performing data ana/vis (analytics/visualization) on the system close to the data source. The data generated by the simulation vary in size and content during execution, and as a result, the time taken by the data ana/vis components of the workflow also change and can significantly impact the performance of the workflow. Managing these dynamics requires adapting workflow configuration and optimizing its execution at runtime. This thesis explores a data-driven approach to enable this runtime adaptation of the in-situ workflow execution and develops programming abstractions as well as runtime mechanics. Firstly, the data service used for managing the in-situ workflow and processing the data analytics pipeline needs to be adaptable according to the content of the simulation data. For example, the data service needs to detect interesting data based on user-defined functions and trigger particular data analytics when interesting scientific phenomenons happen. Besides, the variation of workflow elements may influence proper in-situ design patterns and how to place the data ana/vis task such as the loosely coupled or tightly coupled way. The proper decisions are influenced by multiple workflow runtime metrics such as the ana/vis execution time, size of the data, and the workload of the data staging service. It is challenging to decide how these parameters influence the in-situ design patterns and choose a proper pattern at runtime. Furthermore, the variation of the data size generated by the simulation may also influence the requirements of computing resources used for in-situ processing. The in-situ processing needs to support resource elasticity, i.e., the ability to change resource configurations such as the number of computing nodes/processes during workflow execution, to finish the analytics with time constraints. In order to address the above challenges in optimizing the in-situ workflow, this thesis explores a data-driven programming approach to enable adjusting the workflow configuration during its execution and is developing programming abstractions as well as runtime systems. Specifically, it presents an autonomic approach based on “data-driven triggers” and associated workflow management strategies and applies these concepts to develop multiple optimization scenarios for tuning the in-situ workflow configurations. Explored strategies include the use of the data-driven trigger approach to adapt the data analysis pipeline, adapt data analytics task placement, and control resource elasticity. These presented data-driven triggers-based approaches have been prototyped and experimentally evaluated using synthetic and real application workflows on the current high-end computing system, such as the Cray XC40 system at the National Energy Research Scientific Computing (NERSC).