Building a Virtualized Distributed Computing Infrastructure by Harnessing Grid and Cloud Technologies

In this article, we present the realization of a system, termed as InterGrid, for interconnecting distributed computing infrastructures by harnessing virtual machines. The InterGrid aims to provide an execution environment for running applications on top of the interconnected infrastructures. The system uses virtual machines as the building blocks to construct execution environments that span multiple computing sites. An execution environment is a network of virtual machines created to fulfill the requirements of an application, thus running isolated from other execution environments. These environments can be extended to operate on Cloud infrastructure, such as Amazon EC2. The article provides an abstract view of the proposed architecture and its implementation; experiments show the scalability of an infrastructure managed by InterGrid and how the system can benefit from using Cloud infrastructure. Introduction Over the last decade, the distributed computing area has been characterized by the deployment of large-scale Grids such as EGEE [1] and Grid’5000 [2],. Such Grids have provided the research community with an unprecedented number of resources, which have been used for various scientific endeavors. Several efforts have been made to enable interoperation between Grids by providing standard components and adapters for secure job submissions, data transfers, and information queries [3]. Despite these efforts, the heterogeneity of hardware and software has contributed to increasing complexity of deploying applications on these infrastructures. Moreover, recent advances in virtualization technologies [4] have led to the emergence of commercial infrastructure providers, also known as Cloud computing [5]. Handling the ever growing demands of distributed applications while addressing heterogeneity, remains a challenging task that can require resources from both Grids and clouds. In previous work, we presented an architecture for resource sharing between Grids [6] inspired by the peering agreements established between Internet Service Providers (ISPs) in the Internet, through which ISPs agree to allow traffic into one another's networks. This work presents the realization of the architecture, which is termed as InterGrid and relies on InterGrid Gateways (IGGs) that mediate access to resources of participating Grids. The InterGrid also aims at tackling the heterogeneity of hardware and software within Grids. The use of virtualization technology can ease the deployment of applications spanning multiple Grids as it allows for resource control in a contained manner. In this way, resources allocated by one Grid to another are used to deploy virtual machines. Virtual machines also allow the use of resources from Cloud computing providers. We first introduce the concepts on virtualization and Cloud computing. Then, we describe the InterGrid project (www.gridbus.org/intergrid), which aims to provide an infrastructure for deploying applications on several computing sites, or Grids, by using virtualization technologies. These applications run on networks of virtual machines or execution environments created on top of the physical infrastructure. Next, the system design and implementation are presented. To illustrate the different interactions among the system components, we describe the InterGrid at runtime. Finally, experimental results demonstrate the system scalability and show how applications can combine resources from two cloud provider sites. Background and Context Virtualization Technology and Infrastructure as a Service The increasing ubiquity of Virtual Machine (VM) technologies has enabled the creation of customized environments atop physical infrastructure and the emergence of business models such as Cloud computing. The use of VMs brings several benefits such as: (i) Server consolidation, allowing workloads of several under-utilized servers to be placed in fewer machines; (ii) The ability to create VMs to run legacy code without interfering with other applications' APIs; (iii) Improved security through the creation of sandboxes for running applications with questionable reliability; and (iv) Performance isolation, thus allowing a provider to offer some guarantees and better QoS to customers' applications. Existing virtual-machine based resource management systems can manage a cluster of computers within a site allowing the creation of virtual workspaces [7] or virtual clusters [8]. They can bind resources to virtual clusters or workspaces according to a user's demand. These systems commonly provide an interface through which one can allocate VMs and configure them with the operating system and software of choice. These resource managers, also named Virtual Infrastructure Engines (VIE), allow the user to create customized virtual clusters using shares of the physical machines available at the site. As explained earlier, virtualization technology minimizes some security concerns inherent to the sharing of resources among multiple computing sites. We utilize virtualization software to realize the InterGrid architecture because existing cluster resource managers relying on VMs can provide us with the building blocks, such as the availability information, required for the creation of virtual execution environments. In addition, relying on VMs eases the deployment of execution environments on multiple computing sites; the user application can have better control over the software installed on the resources allocated without compromising the operation of the hosts' operating systems at the computing sites. Virtualization technologies have also facilitated the realization of Cloud Computing services. Cloud computing includes three kinds of services accessible by the Internet: Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). This work considers only IaaS. IaaS aims to give provide computing resources or storage as a service to users. One of the major players in Cloud computing is Amazon with its Elastic Compute Cloud (EC2), which comprises several data centers located around the world. EC2 allows users to deploy VMs on demand on Amazon’s infrastructure and pay only for the computing, storage and network resources they use.