Powering statistical genetics with the grid: using GridWay to automate R-based workflows

Many computationally intense workflows are composed of the same algorithm applied to many data sets. For example, it is good practice in statistical genetics to assess the validity of a method by simulating thousands of datasets of known properties. Further each simulation may involve using permutation tests that necessitate repeating analyses thousands of times per data set. Improvements in the overall throughput of this workflow can be achieved with a straightforward increase in the number of computations that can take place simultaneously. High performance compute clusters have significantly improved the ability to run many such computations simultaneously and have shown the adaptability of these workflows to ever-increasing processing capacity.While clusters of increasing size can be constructed to improve throughput, additional hardware acquisitions impose increasing financial and operating environment burdens. Leveraging multiple clusters in distributed operating centers can dramatically increase capacity while alleviating infrastructure growth burdens, but adds significant complexity in managing workflows across heterogeneous systems and administrative domains.Grid computing can offer immediate benefits in this area of multi-cluster workflows by offering a consistent, full-featured, programmatic interface and uniform identity infrastructure across cluster and administrative boundaries. The Globus Toolkit in combination with the GridWay meta-scheduler more realistically improves the ability to leverage large, geographically distributed, multi-cluster collections to orchestrate workflows for significant gains in throughput.Researchers in the field of biostatistics at UAB are heavy users of the R statistical and graphical software environment, available on multiple clusters on campus. Their R-based statistical analysis workflow falls in the above category of applications. Large numbers of R job runs are currently being managed by manually dividing the workload across clusters based on resources availability. This is clearly cumbersome for the end user to manage and unlikely to result in the optimal division of labor amongst available resources.The focus of our efforts to grid-enable R uses the GridWay meta-scheduler to access existing resources through the Globus-based, campus grid platform, and improve the workflow management across this set of compute resources, optimizing throughput. Because GridWay schedules jobs on multiple clusters using the uniform interface of the Globus Toolkit, this solution promises to transparently increase the throughput for the R workflow with the simple inclusion of additional compute resources. Our plans include adding a large shared-memory compute resource from the state supercomputing center and other clusters through collaborations with partners in SURAgrid, a regional grid infrastructure.There are many dimensions to "grid-enabling" applications, including complex re-engineering of algorithms. It is advantageous, however, to take a step-wise approach to grid adoption that first maximizes workflow throughput by leveraging the most broadly available, commodity infrastructures. By concentrating on throughput gains first, existing infrastructure investments can be maximized and time-consuming algorithmic redesigns can be delayed until more capable infrastructures exist to address coordination and latency considerations. This presentation will focus on this first step of grid-enabling R and detail experiences and performance gains.