A study on the selection of resources from XSEDE supercomputers and the open science grid
Creators: name: Ha, Ming Tai
Publisher: name: No Publisher Supplied
Year: 2018

description: The effective selection of resources on supercomputers and grids improves workload schedul-ing and reduces workload time-to-completion. Lower workload time-to-completions allow scien-tists to gain scientific insights from simulations more quickly. For example, molecular dynamics(MD) simulations are revolutionizing the development of new therapeutics. In the past 6 years,at least 4 billion core-hours consumed on XSEDE machines were from MD software packagesalone. Given the increasing number of MD simulations executed and the increasing amounts ofcomputation they require, there is a need for greater efficiency in resource utilization. Thus, weinvestigate how resources from XSEDE supercomputers and OSG can be effectively selected toreduce workload time-to-completion.Effective resource selection on grids and supercomputers is difficult. Grids have heterogeneousand transient pools of resources, whose availability and performance vary over time. This makesit difficult to collect information, such as benchmarking results, application profiles and hardwarecapabilities, used by techniques like application performance modeling and benchmarking to bestselect grid resources. While the performance of supercomputing resources are easier to assessfor grid resources, the acquisition of these resources requires waiting on a queue, sometimes forlong periods of time. However, accurately predicting queue waiting time remains difficult.In this thesis, we studied how to effectively select resources from XSEDE supercomputers andOSG in the presence of limited information. We developed a formalism that allows us to modelthe cost of task execution based on the information available from XSEDE supercomputers andXSEDE OSG. On the base of our formalism, we constructed the Limited Information Model(LIM) to predict the execution times of compute-intensive, single-threaded, single-process tasks.We evaluated the accuracy of these predictions and the resources selected using these predictionsiito gain insight into what information, if any, would be needed to operate better predictions andresource selections. To overcome the difficulty of selecting resources using queue waiting times,we also developed the resource re-selection process, by which tasks are re-assigned to different,acquired resources at runtime. Resource re-selection uses task execution times and resourceacquisition times, but not queue waiting times. We show that tasks can be effectively re-assignedeven when using inaccurate execution time predictions.Experimental validation of LIM shows that LIM’s predictions are within 157–171% erroron XSEDE supercomputers and 18–31% on OSG. By accounting for the differences in softwareconfigurations on XSEDE supercomputers and OSG, LIM’s predictions can still be used correctlyrank XSEDE supercomputers and OSG. Experiments also show that workloads executed usingresources selected with LIM’s predictions have 67–78% lower workload time-to-completion thanthose executed using randomly selected resources. However, executing workloads on resourcesselected using LIM’s predictions contributed to ∼29–99% of the reduction in workload time-to-completion. We found that the queue waiting times largely influences workload time-to-completion and should also be considered to effectively select resources from supercomputersand grids.Finally, experiments show that performing resource re-selection allows tasks to avoid expe-riencing large queue waiting times on XSEDE supercomputers by executing on OSG instead.Despite using inaccurate task execution time predictions and no queue waiting time predictions,experiments show that resource re-selection reduces task queue waiting times by up to 99%and workload time-to-completion by up to 73% when the queue waiting time experienced onXSEDE supercomputers are high. However, resource re-selection shows little to no reductionsin task queue waiting times and workload time-to-completions when the queue waiting timesexperienced on XSEDE supercomputers are small.