Staging-based data management for extreme scale coupled scientific workflows
Creators: name: Sun, Qian
Publisher: name: No Publisher Supplied
Year: 2017

description: Advanced scientific workflows running at extreme scale on high end computing platforms are providing new capabilities and new opportunities for insights in a wide range of application domain. These workflows compose multiple simulations, data analysis and other application components that require data sharing and exchange at runtime. However, due to the increasing data volumes and associated I/O costs (latency and energy), the traditional disk I/O-base approach for data sharing between the components of these workflows is becoming infeasible. Recently, in memory data staging has emerged as an attractive approach to address these challenges.    The increasing systems scales, the overall complexity of current scientific workflows, as well as the complexity of the coupling and coordination behaviors between workflow components are presenting several new data management challenges that are impacting the effectiveness and efficiency of staging-base solutions. First, query-driven data analysis is an important technique used by scientists to capture intermittent transient information from simulation data, and it is important that a staging-based solution supports this capability. As such queries are performed iteratively and frequently, management of data within the staging solution can be challenging. Second, many of scientific workflows compose multiple concurrently running applications that exhibit distinct data access behaviors and dynamic data exchange patterns. Such dynamic runtime coupling data access behaviors, along with imbalanced data distributions and resource contention, increase the complexity of in-staging data management. Third, many components of the simulation workflows are transitioning to be implemented using task-based runtimes, which employ an asynchronous execution model that provides a finer granularity of control over task execution and enables higher levels of concurrency. However, the decomposition of workflow components into tasks further introduce new in-staging data management challenges.    This thesis addresses these in-staging data management challenges to enable effective and efficient staging-based simulation workflows, and makes the following contributions: (1) Formulates a performance model to capture the fundamental factors that impacting the performance of scientific workflow using data staging. (2) Design of a staging-based scalable indexing and querying framework to enable value-based querying of live simulation data while the simulation is running. (3) Design of an adaptive data placement runtime that leverages application data access patterns and system network topology to dynamically and adaptively place data within a data staging area to reduce data access costs. (4) Design of an adaptive data placement runtime for task-based workflow components that targets coupled application workflows with asynchronous coupling patterns to adaptively place data across the staging cores with awareness of runtime workload and data locality, so as to improve the overall performance of the workflow (in terms of end-to-end execution time and data movement) and/or improve the quality of data processing. The research components have been prototyped and experimentally evaluated using real application workflows on current high end computing systems, including the Hopper, a Cray XE6 system hosted by NERSC and Titan cray XK7 system at Oak Ridge National Laboratory, and their effectiveness and performance are demonstrated.