Abstract:
A cross-host multi-hypervisor system, including a plurality of host sites, each site including at least one hypervisor, each of which includes at least one virtual server, at least one virtual disk that is read from and written to by the at least one virtual server, a tapping driver in communication with the at least one virtual server, which intercepts write requests made by any one of the at least one virtual server to any one of the at least one virtual disk, and a virtual data services appliance, in communication with the tapping driver, which receives the intercepted write requests from the tapping driver, and which provides data services based thereon, and a data services manager for coordinating the virtual data services appliances at the site, and a network for communicatively coupling the plurality of sites, wherein the data services managers coordinate data transfer across the plurality of sites via the network.

Description:
CROSS REFERENCES TO RELATED APPLICATIONS 
       [0001]    This application is a continuation-in-part of U.S. application Ser. No. 13/039,446, entitled METHODS AND APPARATUS FOR PROVIDING HYPERVISOR LEVEL DATA SERVICES FOR SERVER VIRTUALIZATION, filed on Mar. 3, 2011 by inventor Ziv Kedem, which claims priority benefit of U.S. Provisional Application No. 61/314,589, entitled METHODS AND APPARATUS FOR PROVIDING HYPERVISOR LEVEL DATA SERVICES FOR SERVER VIRTUALIZATION, filed on Mar. 17, 2010 by inventor Ziv Kedem. 
     
    
     FIELD OF THE INVENTION 
       [0002]    The present invention relates to virtual server computing environments. 
       BACKGROUND OF THE INVENTION 
       [0003]    Data center virtualization technologies are now well adopted into information technology infrastructures. As more and more applications are deployed in a virtualized infrastructure, there is a growing need for recovery mechanisms to support mission critical application deployment, while providing complete business continuity and disaster recovery. 
         [0004]    Virtual servers are logical entities that run as software in a server virtualization infrastructure, referred to as a “hypervisor”. Examples of hypervisors are VMWARE® ESX manufactured by VMware, Inc. of Palo Alto, Calif., HyperV manufactured by Microsoft Corporation of Redmond, WA, XENSERVER® manufactured by Citrix Systems, Inc. of Fort Lauderdale, Fla., Redhat KVM manufactured by Redhat, Inc. of Raleigh, N.C., and Oracle VM manufactured by Oracle Corporation of Redwood Shores, Calif. A hypervisor provides storage device emulation, referred to as “virtual disks”, to virtual servers. Hypervisor implements virtual disks using back-end technologies such as files on a dedicated file system, or raw mapping to physical devices. 
         [0005]    As distinct from physical servers that run on hardware, virtual servers run their operating systems within an emulation layer that is provided by a hypervisor. Although virtual servers are software, nevertheless they perform the same tasks as physical servers, including running server applications such as database applications, customer relation management applications and MICROSOFT EXCHANGE SERVER®. Most applications that run on physical servers are portable to run on virtual servers. As distinct from virtual desktops that run client side applications and service individual users, virtual servers run applications that service a large number of clients. 
         [0006]    As such, virtual servers depend critically on data services for their availability, security, mobility and compliance requirements. Data services include inter alia continuous data protection, disaster recovery, remote replication, data security, mobility, and data retention and archiving policies. 
         [0007]    Conventional replication and disaster recovery systems were not designed to deal with the demands created by the virtualization paradigm. Most conventional replication systems are not implemented at the hypervisor level, with the virtual servers and virtual disks, but instead are implemented at the physical disk level. As such, these conventional systems are not fully virtualization-aware. In turn, the lack of virtualization awareness creates an operational and administrative burden, and a certain degree of inflexibility. 
         [0008]    It would thus be of advantage to have data services that are fully virtualization-aware. 
       SUMMARY OF THE DESCRIPTION 
       [0009]    Aspects of the present invention relate to a dedicated virtual data service appliance (VDSA) within a hypervisor that can provide a variety of data services. Data services provided by the VDSA include inter alia replication, monitoring and quality of service. The VDSA is fully application-aware. 
         [0010]    In an embodiment of the present invention, a tapping filter driver is installed within the hypervisor kernel. The tapping driver has visibility to I/O requests made by virtual servers running on the hypervisor. 
         [0011]    A VDSA runs on each physical hypervisor. The VDSA is a dedicated virtual server that provides data services; however, the VDSA does not necessarily reside in the actual I/O data path. When a data service processes I/O asynchronously, the VDSA receives the data outside the data path. 
         [0012]    Whenever a virtual server performs I/O to a virtual disk, the tapping driver identifies the I/O requests to the virtual disk. The tapping driver copies the I/O requests, forwards one copy to the hypervisor&#39;s backend, and forwards another copy to the VDSA. 
         [0013]    Upon receiving an I/O request, the VDSA performs a set of actions to enable various data services. A first action is data analysis, to analyze the data content of the I/O request and to infer information regarding the virtual server&#39;s data state. E.g., the VDSA may infer the operating system level and the status of the virtual server. This information is subsequently used for reporting and policy purposes. 
         [0014]    A second action, optionally performed by the VDSA, is to store each I/O write request in a dedicated virtual disk for journaling. Since all I/O write requests are journaled on this virtual disk, the virtual disk enables recovery data services for the virtual server, such as restoring the virtual server to an historical image. 
         [0015]    A third action, optionally performed by the VDSA, is to send I/O write requests to different VDSAs, residing on hypervisors located at different locations, thus enabling disaster recovery data services. 
         [0016]    The hypervisor architecture of the present invention scales to multiple host sites, each of which hosts multiple hypervisors. The scaling flexibly allows for different numbers of hypervisors at different sites, and different numbers of virtual services and virtual disks within different hypervisors. Each hypervisor includes a VDSA, and each site includes a data services manager to coordinate the VSDA&#39;s at the site, and across other sites. 
         [0017]    Embodiments of the present invention enable flexibly designating one or more virtual servers within one or more hypervisors at a site as being a virtual protection group, and flexibly designating one or more hypervisors, or alternatively one or more virtual servers within one or more hypervisors at another site as being a replication target for the virtual protection group. Write order fidelity is maintained for virtual protection groups. A site may comprise any number of source and target virtual protection groups. A virtual protection group may have more than one replication target. The number of hypervisors and virtual servers within a virtual protection group and its replication target are not required to be the same. 
         [0018]    There is thus provided in accordance with an embodiment of the present invention a cross-host multi-hypervisor system, including a plurality of host sites, each site including at least one hypervisor, each of which includes at least one virtual server, at least one virtual disk that is read from and written to by the at least one virtual server, a tapping driver in communication with the at least one virtual server, which intercepts write requests made by any one of the at least one virtual server to any one of the at least one virtual disk, and a virtual data services appliance, in communication with the tapping driver, which receives the intercepted write requests from the tapping driver, and which provides data services based thereon, and a data services manager for coordinating the virtual data services appliances at the site, and a network for communicatively coupling the plurality of sites, wherein the data services managers coordinate data transfer across the plurality of sites via the network. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0019]    The present invention will be more fully understood and appreciated from the following detailed description, taken in conjunction with the drawings in which: 
           [0020]      FIG. 1  is a simplified block diagram of a hypervisor architecture that includes a tapping driver and a virtual data services appliance, in accordance with an embodiment of the present invention; 
           [0021]      FIG. 2  is a simplified data flow chart for a virtual data services appliance, in accordance with an embodiment of the present invention; 
           [0022]      FIG. 3  is a simplified block diagram of a virtual replication system, in accordance with an embodiment of the present invention; 
           [0023]      FIG. 4  is a simplified block diagram of a cross-host multiple hypervisor system that includes data services managers for multiple sites that have multiple hypervisors, in accordance with an embodiment of the present invention; 
           [0024]      FIG. 5  is a user interface screenshot of bi-directional replication of virtual protection groups, in accordance with an embodiment of the present invention; 
           [0025]      FIG. 6  is a user interface screenshot of assignment of a replication target for a virtual protection group, in accordance with an embodiment of the present invention; and 
           [0026]      FIG. 7  is an example an environment for the system of  FIG. 4 , in accordance with an embodiment of the present invention. 
       
    
    
     LIST OF APPENDICES 
       [0027]    Appendix I is an application programming interface for virtual replication site controller web services, in accordance with an embodiment of the present invention; 
         [0028]    Appendix II is an application programming interface for virtual replication host controller web services, in accordance with an embodiment of the present invention; 
         [0029]    Appendix III is an application programming interface for virtual replication protection group controller web services, in accordance with an embodiment of the present invention; 
         [0030]    Appendix IV is an application programming interface for virtual replication command tracker web services, in accordance with an embodiment of the present invention; and 
         [0031]    Appendix V is an application programming interface for virtual replication log collector web services, in accordance with an embodiment of the present invention. 
       DETAILED DESCRIPTION 
       [0032]    Aspects of the present invention relate to a dedicated virtual data services appliance (VDSA) within a hypervisor, which is used to provide a variety of hypervisor data services. Data services provided by a VDSA include inter alia replication, monitoring and quality of service. 
         [0033]    Reference is made to  FIG. 1 , which is a simplified block diagram of a hypervisor architecture that includes a tapping driver and a VDSA, in accordance with an embodiment of the present invention. Shown in  FIG. 1  is a hypervisor  100  with three virtual servers  110 , three virtual disks  120 , an I/O backend  130  and a physical storage array  140 . Hypervisor  100  uses a single physical server, but runs multiple virtual servers  110 . Virtual disks  120  are a storage emulation layer that provide storage for virtual servers  110 . Virtual disks  120  are implemented by hypervisor  100  via I/O backend  130 , which connects to physical disk  140 . 
         [0034]    Hypervisor  100  also includes a tapping driver  150  installed within the hypervisor kernel. As shown in  FIG. 1 , tapping driver  150  resides in a software layer between virtual servers  110  and virtual disks  120 . As such, tapping driver  150  is able to access I/O requests performed by virtual servers  110  on virtual disks  120 . Tapping driver  150  has visibility to I/O requests made by virtual servers  110 . 
         [0035]    Hypervisor  100  also includes a VDSA  160 . In accordance with an embodiment of the present invention, a VDSA  160  runs on a separate virtual server within each physical hypervisor. VDSA  160  is a dedicated virtual server that provides data services via one or more data services engines  170 . However, VDSA  160  does not reside in the actual I/O data path between I/O backend  130  and physical disk  140 . Instead, VDSA  160  resides in a virtual I/O data path. 
         [0036]    Whenever a virtual server  110  performs I/O on a virtual disk  120 , tapping driver  150  identifies the I/O requests that the virtual server makes. Tapping driver  150  copies the I/O requests, forwards one copy via the conventional path to I/O backend  130 , and forwards another copy to VDSA  160 . In turn, VDSA  160  enables the one or more data services engines  170  to provide data services based on these I/O requests. 
         [0037]    Reference is made to  FIG. 2 , which is a simplified data flow chart for a VDSA, in accordance with an embodiment of the present invention. Shown in  FIG. 2  are an I/O receiver  210 , a hash generator  220 , a TCP transmitter  230 , a data analyzer and reporter  240 , a journal manager  250  and a remote VDSA  260 . Remote VDSA  260  resides on different physical hardware, at a possibly different location. 
         [0038]    As shown in  FIG. 2 , I/O receiver  210  receives an intercepted I/O request from tapping driver  150 . VDSA  160  makes up to three copies of the received I/O requests, in order to perform a set of actions which enable the one or more data services engines  170  to provide various services. 
         [0039]    A first copy is stored in persistent storage, and used to provide continuous data protection. Specifically, VDSA  160  sends the first copy to journal manager  250 , for storage in a dedicated virtual disk  270 . Since all I/O requests are journaled on virtual disk  270 , journal manager  250  provides recovery data services for virtual servers  110 , such as restoring virtual servers  110  to an historical image. In order to conserve disk space, hash generator  220  derives a one-way hash from the I/O requests. Use of a hash ensures that only a single copy of any I/O request data is stored on disk. 
         [0040]    An optional second copy is used for disaster recovery. It is sent via TCP transmitter  230  to remote VDSA  260 . As such, access to all data is ensured even when the production hardware is not available, thus enabling disaster recovery data services. 
         [0041]    An optional third copy is sent to data analyzer and reporter  240 , which generates a report with information about the content of the data. Data analyzer and reporter  240  analyzes data content of the I/O requests and infers information regarding the data state of virtual servers  110 . E.g., data analyzer and reporter  240  may infer the operating system level and the status of a virtual server  110 . 
         [0042]    Reference is made to  FIG. 3 , which is a simplified block diagram of a virtual replication system, in accordance with an embodiment of the present invention. Shown in  FIG. 3  is a protected site designated Site A, and a recovery site designated Site B. Site A includes a hypervisor  100 A with three virtual servers  110 A- 1 ,  110 A- 2  and  110 A- 3 , and a VDSA  160 A. Site A includes two physical disks  140 A- 1  and  140 A- 2 . Site B includes a hypervisor  100 B with a VDSA  160 B. Site B includes two physical disks  140 B- 1  and  140 B- 2 . All or some of virtual servers  110 A- 1 ,  110 A- 2  and  110 A- 3  may be designated as protected. Once a virtual server is designated as protected, all changes made on the virtual server are replicated at the recovery site. 
         [0043]    In accordance with an embodiment of the present invention, every write command from a protected virtual server in hypervisor  100 A is intercepted by tapping driver  150  ( FIG. 1 ) and sent asynchronously by VDSA  160 A to VDSA  160 B for replication, via a wide area network (WAN)  320 , while the write command continues to be processed by the protected server. 
         [0044]    At Site B, the write command is passed to a journal manager  250  ( FIG. 2 ), for journaling on a Site B virtual disk  270  ( FIG. 2 ). After every few seconds, a checkpoint is written to the Site B journal, and during a recovery one of the checkpoints may be selected for recovering to that point. Additionally, checkpoints may be manually added to the Site B journal by an administrator, along with a description of the checkpoint. E.g., a checkpoint may be added immediately prior to an event taking place that may result in the need to perform a recovery, such as a planned switch over to an emergency generator. 
         [0045]    In addition to write commands being written to the Site B journal, mirrors  110 B- 1 ,  110 B- 2  and  110 B- 3  of the respective protected virtual servers  110 A- 1 ,  110 A- 2  and  110 A- 3  at Site A are created at Site B. The mirrors at Site B are updated at each checkpoint, so that they are mirrors of the corresponding virtual servers at Site A at the point of the last checkpoint. During a failover, an administrator can specify that he wants to recover the virtual servers using the latest data sent from the Site A. Alternatively the administrator can specify an earlier checkpoint, in which case the mirrors on the virtual servers  110 B- 1 ,  110 -B- 2  and  110 B- 3  are rolled back to the earlier checkpoint, and then the virtual servers are recovered to Site B. As such, the administrator can recover the environment to the point before any corruption, such as a crash or a virus, occurred, and ignore the write commands in the journal that were corrupted. 
         [0046]    VDSAs  160 A and  160 B ensure write order fidelity; i.e., data at Site B is maintained in the same sequence as it was written at Site A. Write commands are kept in sequence by assigning a timestamp or a sequence number to each write at Site A. The write commands are sequenced at Site A, then transmitted to Site B asynchronously, then reordered at Site B to the proper time sequence, and then written to the Site B journal. 
         [0047]    The journal file is cyclic; i.e., after a pre-designated time period, the earliest entries in the journal are overwritten by the newest entries. 
         [0048]    It will be appreciated by those skilled in the art that the virtual replication appliance of the present invention operates at the hypervisor level, and thus obviates the need to consider physical disks. In distinction, conventional replication systems operate at the physical disk level. Embodiments of the present invention recover write commands at the application level. Conventional replication systems recover write commands at the SCSI level. As such, conventional replication systems are not fully application-aware, whereas embodiment of the present invention are full application-aware, and replicate write commands from an application in a consistent manner. 
         [0049]    The present invention offers many advantages. 
         [0050]    Hardware Agnostic: Because VDSA  160  manages recovery of virtual servers and virtual disks, it is not tied to specific hardware that is used at the protected site or at the recovery site. The hardware may be from the same vendor, or from different vendors. As long as the storage device supports the iSCSI protocol, any storage device, known today or to be developed in the future, can be used. 
         [0051]    Fully Scalable: Because VDSA  160  resides in the hypervisor level, architectures of the present invention scale to multiple sites having multiple hypervisors, as described hereinbelow with reference to  FIG. 4 . 
         [0052]    Efficient Asynchronous Replication: Write commands are captured by VDSA  160  before they are written to a physical disk at the protected site. The write commands are sent to the recovery site asynchronously, and thus avoid long distance replication latency. Moreover, only delta changes are sent to the recovery site, and not a whole file or disk, which reduces the network traffic, thereby reducing WAN requirements and improving recovery time objective and recovery point objective. 
         [0053]    Control of Recovery: An administrator controls when a recovery is initiated, and to what point in time it recovers. 
         [0054]    Near-Zero Recovery Point Objective (RPO): VDSA  160  continuously protects data, sending a record of every write command transacted at the protected site to the recovery site. As such, recovery may be performed within a requested RPO. 
         [0055]    Near-Zero Recovery Time Objective (RTO): During recovery the mirrors of the protected virtual servers are recovered at the recovery site from VDSA  160 B, and synchronized to a requested checkpoint. In accordance with an embodiment of the present invention, during synchronization and while the virtual servers at the recovery site are not yet fully synchronized, users can nevertheless access the virtual servers at the recovery site. Each user request to a virtual server is analyzed, and a response is returned either from the virtual server directly, or from the journal if the information in the journal is more up-to-date. Such analysis of user requests continues until the recovery site virtual environment is fully synchronized. 
         [0056]    WAN Optimization between Protected and Recovery Sites: In accordance with an embodiment of the present invention, write commands re compressed before being sent from VDSA  160 A to VDSA  160 B, with throwing used to prioritize network traffic. As such, communication between the protected site and the recovery site is optimized. 
         [0057]    WAN Failover Resilience: In accordance with an embodiment of the present invention, data is cached prior to being transmitted to the recovery site. If WAN  320  goes down, the cached data is saved and, as soon as WAN  320  comes up again, the data is sent to the recovery site and both sites are re-synchronized. 
         [0058]    Single Point of Control: In accordance with an embodiment of the present invention, both the protected and the recovery site are managed from the same client console. 
         [0059]    As indicated hereinabove, the architecture of  FIG. 1  scales to multiple sites having multiple hypervisors. Reference is made to  FIG. 4 , which is a simplified block diagram of a cross-host multiple hypervisor system  300  that includes data services managers for multiple sites that have multiple hypervisors, in accordance with an embodiment of the present invention. The architecture of  FIG. 4  includes three sites, designated Site A, Site B and Site C, the three sites being communicatively coupled via a network  320 . Each site includes one or more hypervisors  100 . Specifically, Site A includes three hypervisors,  100 A/ 1 ,  100 A/ 2  and  100 A/ 3 , Site B includes two hypervisors,  100 B/ 1  and  100 B/ 2 , and Site C includes one hypervisor  100 C/ 1 . The sites have respective one or more physical disks  140 A,  140 B and  140 C. 
         [0060]    The hypervisors are shown in system  300  with their respective VDSA&#39;s  160 A/ 1 ,  160 A/ 2 , . . . , and the other components of the hypervisors, such as the virtual servers  110  and virtual disks  120 , are not shown for the sake of clarity. An example system with virtual servers  110  is shown in  FIG. 7 , and described hereinbelow. 
         [0061]    The sites include respective data services managers  310 A,  310 B and  310 C that coordinate hypervisors in the sites, and coordinate hypervisors across the sites. 
         [0062]    The system of  FIG. 4  may be used for data replication, whereby data at one site is replicated at one or more other sites, for protection. The solid communication lines  330  in  FIG. 4  are used for in-site traffic, the dashed communication lines  340  are used for replication traffic between sites, and the dotted communication lines  350  are used for control traffic between data services managers. 
         [0063]    Data services managers  310 A,  310 B and  310 C are control elements. The data services managers at each site communicate with one another to coordinate state and instructions. The data services managers track the hypervisors in the environment, and track health and status of the VDSAs  160 A/ 1 ,  160 A/ 2 , . . . . 
         [0064]    It will be appreciated by those skilled in the art that the environment shown in  FIG. 4  may be re-configured by moving one or more virtual servers  110  from one hypervisor  100  to another, by moving one or more virtual disks  120  from one hypervisor  100  to another, and by adding one or more additional virtual servers  110  to a hypervisor  100 . 
         [0065]    In accordance with an embodiment of the present invention, the data services managers enable designating groups of specific virtual servers  110 , referred to as virtual protection groups, to be protected. For virtual protection groups, write order fidelity is maintained. The data services managers enable designating a replication target for each virtual protection group; i.e., one or more sites, and one or more hypervisors in the one or more sites, at which the virtual protection group is replicated. A virtual protection group may have more than one replication target. The number of hypervisors and virtual servers within a virtual protection group and its replication target are not required to be the same. 
         [0066]    Reference is made to  FIG. 5 , which is a user interface screenshot of bi-directional replication of virtual protection groups, in accordance with an embodiment of the present invention. Shown in  FIG. 4  are virtual protection groups  301  (“Exchange”),  302  (“WebApp”),  303  (“Dummy-R1”),  304  (“Windows 2003”) and  305  (Dummies-L”). Arrows  306  indicate direction of replication. 
         [0067]    Reference is made to  FIG. 6 , which is a user interface screenshot of assignment of a replication target for a virtual protection group, in accordance with an embodiment of the present invention. Shown in  FIG. 6  is an entry  307  for designating a recovery host, and an entry  308  for designating a recovery datastore for virtual protection group  304  (“Windows 2003”) of  FIG. 5 . Respective source and target datastores, [SAN ZeRTO-30]  309 A and [datastore1]  309 B, are shown as being paired. 
         [0068]    More generally, the recovery host may be assigned to a cluster, instead of to a single hypervisor, and the recovery datastore may be assigned to a pool of resources, instead of to a single datastore. Such assignments are of particular advantage in providing the capability to recover data in an enterprise internal cloud that includes clusters and resource pools, instead of using dedicated resources for recovery. 
         [0069]    The data services managers synchronize site topology information. As such, a target site&#39;s hypervisors and datastores may be configured from a source site. 
         [0070]    Virtual protection groups enable protection of applications that run on multiple virtual servers and disks as a single unit. E.g., an application that runs on virtual servers many require a web server and a database, each of which run on a different virtual server than the virtual server that runs the application. These virtual servers may be bundled together using a virtual protection group. 
         [0071]    Referring back to  FIG. 4 , data services managers  310 A,  310 B and  310 C monitor changes in the environment, and automatically update virtual protection group settings accordingly. Such changes in the environment include inter alia moving a virtual server  110  from one hypervisor  100  to another, moving a virtual disk  120  from one hypervisor  100  to another, and adding a virtual server  110  to a hypervisor  100 . 
         [0072]    For each virtual server  110  and its target host, each VDSA  160 A/ 1 ,  160 A/ 2 , . . . replicates IOs to its corresponding replication target. The VDSA can replicate all virtual servers to the same hypervisor, or to different hypervisors. Each VDSA maintains write order fidelity for the IOs passing through it, and the data services manager coordinates the writes among the VDSAs. 
         [0073]    Since the replication target hypervisor for each virtual server  110  in a virtual protection group may be specified arbitrarily, all virtual servers  110  in the virtual protection group may be replicated at a single hypervisor, or at multiple hypervisors. Moreover, the virtual servers  110  in the source site may migrate across hosts during replication, and the data services manager tracks the migration and accounts for it seamlessly. 
         [0074]    Reference is made to  FIG. 7 , which is an example an environment for system  300 , in accordance with an embodiment of the present invention. As shown in  FIG. 7 , system  300  includes the following components. 
       Site A 
       [0075]    Hypervisor  100 A/ 1 : virtual servers  110 A/ 1 - 1 ,  110 A/ 1 - 2 ,  110 A/ 1 - 3 .
 
Hypervisor  100 A/ 2 : virtual servers  110 A/ 2 - 1 ,  110 A/ 2 - 2 ,  110 A/ 2 - 3 .
 
Hypervisor  100 A/ 3 : virtual servers  110 A/ 3 - 1 ,  110 A/ 3 - 2 ,  110 A/ 3 - 3 .
 
       Site B 
       [0076]    Hypervisor  100 B/ 1 : virtual servers  110 B/ 1 - 1 ,  110 B/ 1 - 2 ,  110 B/ 1 - 3 .
 
Hypervisor  100 B/ 2 : virtual servers  110 B/ 2 - 1 ,  110 B/ 2 - 2 ,  110 B/ 2 - 3 .
 
       Site C 
       [0077]    Hypervisor  100 C/ 1 : virtual servers  110 C/ 1 - 1 ,  110 C/ 1 - 2 ,  110 C/ 1 - 3 ,  110 C/ 1 - 4 . 
         [0078]    As further shown in  FIG. 7 , system  300  includes the following virtual protection groups. Each virtual protection group is shown with a different hatching, for clarity. 
         [0000]    VPG 1  (shown with upward-sloping hatching) 
         [0079]    Source at Site A: virtual servers  110 A/ 1 - 1 ,  110 A/ 2 - 1 ,  110 A/ 3 - 1   
         [0080]    Replication Target at Site B: virtual servers  110 B/ 1 - 1 ,  110 B/ 1 - 2 ,  110 B/ 2 - 1   
         [0000]    VPG 2  (shown with downward-sloping hatching) 
         [0081]    Source at Site B: virtual servers  110 B/ 1 - 3 ,  110 B/ 2 - 2   
         [0082]    Replication Target at Site A: virtual servers  110 A/ 1 - 2 ,  110 A/ 2 - 2   
         [0000]    VPG 3  (shown with horizontal hatching) 
         [0083]    Source at Site A: virtual server  110 A/ 3 - 3   
         [0084]    Replication Target at Site B: virtual serer  110 B/ 2 - 3   
         [0085]    Replication Target at Site C: virtual server  110 C/ 1 - 4   
         [0000]    VPG 4  (shown with vertical hatching) 
         [0086]    Source at Site A: virtual servers  110 A/ 1 - 3 ,  110 A/ 2 - 3 ,  110 A/ 3 - 2   
         [0087]    Replication Target at Site C: virtual servers  110 C/ 1 - 1 ,  110 C/ 1 - 2 ,  110 C/ 1 - 3   
         [0088]    As such, it will be appreciated by those skilled in the art that the hypervisor architecture of  FIG. 1  scales to multiple host sites, each of which hosts multiple hypervisors. The scaling flexibly allows for different numbers of hypervisors at different sites, and different numbers of virtual services and virtual disks within different hypervisors. 
         [0089]    The present invention may be implemented through an application programming interface (API), exposed as web service operations. Reference is made to Appendices I-V, which define an API for virtual replication web services, in accordance with an embodiment of the present invention. 
         [0090]    In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will, however, be evident that various modifications and changes may be made to the specific exemplary embodiments without departing from the broader spirit and scope of the invention as set forth in the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.