Complexity Analysis and Performance Optimization of Distributed Computing Workflows : From Theory to Practice

The advance of supercomputing technology is expediting the transition in various basic and applied sciences from traditional laboratory-controlled experimental methodologies to modern computational paradigms involving complex numerical model analyses and extreme-scale simulations. These computationbased simulations and analyses have become an essential research and discovery tool in next-generation scientific applications and are producing colossal amounts of simulation data. Other scientific data of similar scales generated in broad science communities include real-world environmental observations such as satellite climate data [1] and multimodal sensor data, and high-throughput experimental measurements such as Spallation Neutron Source [2] and Large Hadron Collider [3]. No matter which type of scientific data is considered, one crucial task would be to develop an effective and efficient endto-end solution for geographically distributed users to transfer, process, visualize, analyze, and synthesize the data in heterogeneous network environments for collaborative research [4], [5]. The computing tasks of these applications feature complex workflows consisting of many computing modules with intricate inter-module dependencies and require the use of a wide range of expensive resources including supercomputers, PC clusters, high-end workstations, and experimental facilities [6]. Typically, these resources are deployed at various research institutes and national laboratories, and are provided to application users through wide-area network connections that may span through several countries [2], [7], hence inevitably exhibiting an inherent dynamic nature in their accessibility, availability, capacity, and reliability. As new computing and networking technologies rapidly emerge, enabling functionalities are progressing at an ever-increasing pace, unfortunately, so are the dynamics, scale, heterogeneity, and complexity of the networked computing environments. Application users need to manually configure and run computing tasks over networks on their own, oftentimes resulting in unsatisfactory performance in such diverse and dynamic environments. As thus, the science community has identified an urgent need to support such distributed workflows and optimize their end-to-end performance in order to ensure the success of mission-critical e-science and maximize the utilization of massively distributed resources. We investigate the problem of mapping computing workflows structured as either linear pipelines or Directed Acyclic Graphs (DAGs) to heterogeneous network environments with the goal of automating workflow executions and optimizing their endto-end performance in terms of Minimum End-to-end Delay (MED) for unitary-input applications (i.e. one-time processing) and Maximum Frame Rate (MFR) for serial-feed applications (i.e. streaming processing). We construct rigorous cost models for workflows and networks, and formulate the workflow mapping as a set of optimization problems with different objectives and mapping constraints. For each of these problems, we conduct in-depth analysis of computational complexity and develop either an optimal or an efficient heuristic solution [8]– [14]. The performance superiority of the proposed approaches is illustrated by extensive simulation-based comparisons with existing algorithms and further verified by large-scale experiments on real-life scientific workflows through effective system implementation and deployment in real networks [15], [16].