A novel probabilistic temporal framework and its strategies for cost-effective delivery of high QoS in scientific cloud workflow systems
Creators: name: Liu, Xiaji
Publisher: name: Swinburne
Year: 2024

description: Cloud computing is a latest market-oriented computing paradigm which can provide virtually unlimited scalable high performance computing resources. As a type of high-level middleware services for cloud computing, cloud workflow systems are a research frontier for both cloud computing and workflow technologies. Cloud workflows often underlie many large scale data/computation intensive e-science applications such as earthquake modelling, weather forecast and Astrophysics. At build-time modelling stage, these sophisticated processes are modelled or redesigned as cloud workflow specifications which normally contain the functional requirements for a large number of workflow activities and their non-functional requirements such as Quality of Service (QoS) constraints. At runtime execution stage, cloud workflow instances are executed by employing the supercomputing and data sharing ability of the underlying cloud computing infrastructures. In this thesis, we focus on scientific cloud workflow systems. In the real world, many scientific applications need to be time constrained, i.e. they are required to be completed by satisfying a set of temporal constraints such as local temporal constraints (milestones) and global temporal constraints (deadlines). Meanwhile, task execution time (or activity duration), as one of the basic measurements for system performance, often needs to be monitored and controlled by specific system management mechanisms. Therefore, how to ensure satisfactory temporal correctness (high temporal QoS), i.e. how to guarantee on-time completion of most, if not all, workflow applications, is a critical issue for enhancing the overall performance and usability of scientific cloud workflow systems. At present, workflow temporal verification is a key research area which focuses on time-constrained large-scale complex workflow applications in distributed high performance computing environments. However, existing studies mainly emphasise on monitoring and detection of temporal violations (i.e. violations of temporal constraints) at workflow runtime, there is still no comprehensive framework which can support the whole lifecycle of time-constrained workflow applications in order to achieve high temporal QoS. Meanwhile, cloud computing adopts a marketoriented resource model, i.e. cloud resources such as computing, storage and network are charged by their usage. Hence, the cost for supporting temporal QoS (including both time overheads and monetary cost) should be managed effectively in scientific cloud workflow systems. This thesis proposes a novel probabilistic temporal framework and its strategies for cost-effective delivery of high QoS in scientific cloud workflow systems (or temporal framework for short in this thesis). By investigating the limitations of conventional temporal QoS related research, our temporal framework can provide a systematic and cost-effective support for time-constrained scientific cloud workflow applications over their whole lifecycles. With a probability based temporal consistency model, there are three major components in the temporal framework: Component 1 – temporal constraint setting; Component 2 – temporal consistency monitoring; Component 3 – temporal violation handling. Based on the investigation and analysis, we propose some new concepts and develop a set of innovative strategies and algorithms towards cost-effective delivery of high temporal QoS in scientific cloud workflow systems. Case study, comparisons, quantitative evaluations and/or mathematical proofs are presented for the evaluation of each component. These demonstrate that our new concepts, innovative strategies and algorithms for the temporal framework can significantly reduce the cost for the detection and handling of temporal violations while achieving high temporal QoS in scientific cloud workflow systems. Specifically, at scientific cloud workflow build time, in Component 1, a statistical time-series pattern based forecasting strategy is first conducted to predict accurate duration intervals of scientific cloud workflow activities. Afterwards, based on the weighted joint normal distribution of workflow activity durations, a probabilistic setting strategy is applied to assign coarse-grained temporal constraints through a negotiation process between service users and service providers, and then fine-grained temporal constraints can be propagated along scientific cloud workflows in an automatic fashion. At scientific cloud workflow runtime, in Component 2, the state of scientific cloud workflow execution towards specific temporal constraints, i.e. temporal consistency, is monitored constantly with the following two steps: first, a minimum probability time redundancy based temporal checkpoint selection strategy determines the workflow activities where potential temporal violations take place; second, according to the probability based temporal consistency model, temporal verification is conducted on the selected checkpoints to check the current temporal consistency states and the type of temporal violations. In Component 3, detected temporal violations are handled with the following two steps: first, an adaptive temporal violation handling point selection strategy decides whether a temporal checkpoint should be selected as a temporal violation handling point to trigger temporal violation handling strategies; Second, at temporal violation handling points, different temporal violation handling strategies are executed accordingly to tackle different types of temporal violations. In our temporal framework, we focus on metaheuristics based workflow rescheduling strategies for handling statistically recoverable temporal violations. The major contributions of this research are that we have proposed a novel comprehensive temporal framework which consists of a set of new concepts, innovative strategies and algorithms for supporting time-constrained scientific applications over their whole lifecycles in cloud workflow systems. With these, we can significantly reduce the cost for detection and handling of temporal violations whilst delivering high temporal QoS in scientific cloud workflow systems. This would eventually improve the overall performance and usability of cloud workflow systems because a temporal framework can be viewed as a software service for cloud workflow systems. Consequently, by deploying the new concepts, innovative strategies and algorithms, scientific cloud workflow systems would be able to better support large-scale sophisticated e-science applications in the context of cloud economy.