Private cloud replication and recovery

Replication and recovery for a protected private cloud infrastructure that may include hosts, virtual machines (VMs) provisioned on the hosts, storage arrays and a management server. Metadata is periodically captured and made accessible to a recovery site. Upon a recovery event, replication of storage arrays is halted, and a number of target machines corresponding to the management server and the hosts to be recovered are assigned. The assigned management server and hosts are then bare provisioned by installing operating systems or hypervisors as specified by the metadata. Only then are recovery target machines connected to the replicated storage arrays so that virtual machines can be activated.

BACKGROUND

Replication of data processing systems to maintain operational continuity is now required almost everywhere. The costs incurred during downtime when information technology equipment and services are not available can be significant, and sometimes even cause an enterprise to halt operations completely. With replication, aspects of data processing machines that may change rapidly over time, such as their program and data files, physical volumes, file systems, etc. can be duplicated on a scheduled or continuous basis. Replication may be used for many purposes such as assuring data availability upon equipment failure, site disaster recovery or planned maintenance operations.

Replication may be directed to either the physical or virtual processing environment and/or different abstraction levels. For example, one may undertake to is replicate each physical machine exactly as it exists at a given time. However, replication processes may also be architected along virtual data processing lines, with corresponding virtual replication processes, with the end result being to remove the physical boundaries and limitations associated with particular physical machines.

Use of a replication service as provided by a remote or hosted external service provider can have numerous advantages. Replication services can provide continuous availability and failover capabilities that are more cost effective than an approach which has the data center operator owning, operating and maintaining a complete suite of duplicate machines at its own data center. With such replication services, physical or virtual machine infrastructure is replicated at a remote and secure data center.

In the case of replication services to virtual target, a virtual disk file containing the processor type and configuration, operating system, data, and applications for each data processor in the production environment is created and retained in a dormant state. In the event of a disaster, the virtual disk file is moved to a production mode within a remote and secure data center. Applications and data can then be accessed on the remote data center, enabling the service customer to continue operating from the cloud while recovering from a disaster.

From the perspective of the service customer, the replication service provider thus offers a Recover to Cloud (R2C) service that is provided much like an on-demand utility (similar to the electricity grid) over a network (typically the Internet). This is enables a data center operator to replicate critical servers and applications in his production environment to the cloud.

Therefore, existing disaster recovery products do accommodate virtualized environments. They can also provide centralized management of recovery plans enabling non-destructive testing and automated site recovery and migration processes. These products can also be used to specify which data process resources are to be recovered. However, such products most often require provisioning of resources at the recovery site in advance of a recovery event and do not offer optimum flexibility.

SUMMARY OF PREFERRED EMBODIMENTS

In a preferred configuration, a protected private cloud infrastructure may include hosts, and virtual machines provisioned on the hosts. This environment may also include storage arrays and a management server. The host machines provide processing resources and memory to the virtual machines. The storage, typically provided by separate hardware, contains an array of disks that may preferably be formed as a storage array network that connects the host machines to the storage arrays. A management server is also responsible for orchestration of the environment as well as maintaining metadata about the components that make up the private cloud virtual infrastructure.

A preferred process operates the management server to permit a user to configure the hosts and storage arrays and to provision virtual machines on the hosts. A separate process continuously replicates the storage arrays to a recovery site. This process may be carried out as part of a continuous storage replication scheme that is operates entirely within the context of the storage array network and separate from the disaster recovery functions.

Metadata is periodically obtained from the management server. The collected metadata configuration of the hosts and virtual machines is replicated at a metadata repository accessible at or located on the recovery site. Thus, it is understood that in a preferred arrangement replicating this metadata occurs independently of replicating the storage arrays.

Upon a recovery event, such as may occur upon disaster or disaster test, replication of the storage arrays is halted. At this point, a number of target machines corresponding to the management server and the hosts are assigned. It should be noted that these hosts are not previously assigned to this task prior to the disaster event.

The assigned management server and hosts are then bare metal provisioned for example, by installing operating systems or hypervisors as specified by the metadata. The management server is then recovered from the metadata to one of the recovery target machines and the hosts are also recovered from the metadata.

It is only after the management server and all such hosts are bare metal provisioned is the next step taken of connecting the recovery target machines to the replicated storage arrays. This causes virtual machine's metadata to also be retrieved for the recovery target machines.

However, prior to accessing the metadata to determine recovery of the virtual machines, the user may access the management server to specify which virtual machines are to actually be recovered. Specification of which virtual machines are to be recovered can therefore be delayed until such time as disaster occurs and the user need not specify which virtual machines are to be recovered in advance of such an event.

This approach to private cloud replication provides distinct advantages from the perspective of a service provider. The physical machines necessary for providing the recovery service need not be specifically tied to any particular protected environment prior to a disaster event. It is only when a customer acquires a need for them, are such is recovery machines tied down to a given role. All that is needed to be persisted by the recovery service is the host and management server metadata. This can be metadata can be stored in a repository, such as a shared database, that provides secure access to different customers.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Introduction

The present disclosure describes a private cloud replication service designed to provide continuous protection for a virtual machine infrastructure. There are two parts to the replication, virtual machine storage and metadata, each describing different aspects of the infrastructure. Though it is not a requirement, storage replication is continuous—data is replicated as it is created. Metadata is replicated periodically and separate from storage replication. At time of recovery automated processes use replicated virtual machine storage and virtual infrastructure metadata to recreate a private cloud in a recovery site.

Typical Private Cloud

FIG. 1illustrates a typical private cloud infrastructure100supported by this service consists of virtual machine hosts104-A,104-B,104-C, storage arrays106-1, is106-2, and a management server102. Hosts (physical machines)104run hypervisor software, which enables multiple virtual machines to run on the same physical hardware. The host machines104provide CPU and memory resources to the (guest) virtual machines. Storage is usually provided by separate hardware, containing an array106of disks. The arrays106can be arranged as a storage area network, or SAN, which is responsible for connecting the host machines to the storage arrays106.

The Management Server102is primarily responsible for orchestration of the private cloud100. It also maintains metadata about the components that make up the private cloud virtual infrastructure100in a manner to be described in more detail below.

The private cloud virtualized environment may be provided using platform such as VMWare. In such an environment, the management server may be based on a VCenter. However, other virtualization solutions may be used.

The more detailed partial view ofFIG. 2shows a representative host, Host A104-A, and the two storage arrays106-1,106-2. Host A is physically connected to both Storage Arrays, represented by solid lines. In this example, Host A is providing CPU and memory to three virtual machines, VM1110-1, VM2110-2, and VM3110-3. Storage Array1160-1is providing storage to VM1110-1and VM2110-2, while Storage Array2160-2is providing storage to VM3110-3. The virtual machines110are also logically connected to their associated virtual disks112-1,112-2,112-3residing on the storage arrays, represented by the dashed lines.

Private Cloud Replication

FIG. 3illustrates the private cloud100as installed at a protected site300. A recovery site350is also shown. Recovery site may typically be operated as a service to the owner/operator of the protected site300. As such, the recovery site350may service many protected site300, although only one protected site300is shown. Under normal operation, the storage arrays106are continuously replicated from the protected site300to the recovery site350. This replication may occur within the content of the is SAN and separately from any recovery process described here.

Because the virtual machines110store their data on the storage arrays106-1,106-2, by way of their virtual disks112, this continuous replication process also automatically and without further intervention by management server102replicates the virtual machine data. As shown inFIG. 3, the recovery site350would be equipped with permanent recovery storage arrays106-R-1,106-R-2to accept the replicated data. An alternative configuration could have no permanently installed storage on the recovery side. Instead, the replication storage106-R arrays could be populated at the time of need by physically bringing data to the recovery site350. Tape based backups would be one such example of a medium for transferring data to the recovery site in a non-continuous fashion.

Metadata Capture

FIG. 4shows how a separate, but simultaneous process handles metadata replication. The storage by itself only contains virtual disks112, or storage for the virtual machines110. Metadata is required to reconstruct the hosts104and to specify the virtual machines110that run inside them. Metadata replication is handled by a capture tool410, which exports data from the management server102and serializes it to an metadata XML document, which can be read at a later time. This metadata replication process is periodic; it typically occurs on a similar schedule as a backup process for example, once per day, late at night. It therefore occurs preferably asynchronous to, and separate from, any changes the user might make to the VM configuration during the day. The XML document containing the metadata would then be transferred to, and persisted by, a metadata repository450located at or at least separately accessible by the recovery site350.

Metadata replication is first handled by a capture tool410, which exports data from the management server102and serializes it to stored form (such as an XML document) which can be read at a later time.

The capture tool410is pre-configured with a list of data elements (metadata), is which describe the private cloud infrastructure. This list is a subset of, but not all of, the available data concerning the private cloud infrastructure. The subset is limited to that data which must be retrieved in order to successfully reconstitute a functional copy of the original private cloud infrastructure at time of test or recovery. The capture tool410makes use of application programming interfaces (APIs) provided by the management server102to create this a subset profile of the private cloud infrastructure. For example, the capture tool can query the management server102for all of the host machines104under its control, and then present that list via the API for generating the subset. It will also look for arrangements of hosts104, such as in clusters or hierarchical folder structures—it may be necessary to make several follow-up queries to get a complete listing of hosts104under the management server's102control.

Once a comprehensive list of hosts and host arrangements are generated, the capture tool410then inspects each of the hosts104for further data. For example, hosts104may contain a number of guest machines and may also contain virtual network devices, storage, or other data center resources111(as described inFIG. 2and shown inFIG. 4, for example). The capture tool410will thus make further inquiries based on what it finds within each host104. Each such guest and/or virtual network device will contain further relevant data, which will be captured during these later requests to the management server API. The metadata is therefore possibly further processed by metadata capture tool410.

At the conclusion of the capture process, the capture tool410operates to construct a document (XML), which is used to transfer the metadata to the recovery site350.

XML File with Virtual Infrastructure Metadata

FIG. 5illustrates a sample of captured virtual infrastructure metadata XML file500contain the information about all of the virtual machines110and virtual machine hosts104and they way they are interconnected at the protected site300. The is sample illustrates the collected metadata.501Information about the management server102: address, datacenter name502Clusters are groupings of hosts104and virtual machines110, used for fault tolerance and virtual machine movement.503Hardware information about a host machine104.504The type of host104. In this example, the host is VMware ESXi.505Network configuration for the host104.506Virtual Switch (running within the host) configuration507Physical network interfaces, used by the virtual switch to route network traffic outside of the host104.508Virtual network interface configuration—used by management network traffic.509Network configuration for the virtual network interface.510Types of traffic, which can flow over the virtual network interface.511Port Groups are groupings of virtual ports on a virtual switch.512Physical network interface configuration; referenced earlier in line7.513Virtual Machine (or guest) definitions.514Resource Pool definitions. Resource pools are used to distribute CPU and Memory resources.

It should be understood this is a simplified example and an actual metadata file would typically include more information that is not initial to recovery.

Private Cloud Recovery

As shown inFIG. 6, time of recovery, the replication between storage devices is broken. This can be planned, in the case if a test, or can be unplanned as a result of a failure at the protected site (e.g., one or more connections in the SAN are broken).

Turning attention toFIG. 7, a preferred recovery process begins by pulling metadata from the repository450and matching available physical machines in the recovery site350to physical machines that existed on the protected site. Sample is mapping:

This mapping can be represented by another file. One example for such file710XML mapping the management server102and Host A104-A, is shown below. It shows that role of management server102will be given to recovery_D610-S and the recovery_E610-E will take the role of Host A104-A. This also shows the deploy flag, which is set to true by default, meaning that Host A will be recovered.

This mapping is then used by automated processes to install software and configure the target hardware610, as needed. Recovery target servers610can be managed as a pool of general resources, and allocated to recovery of a specific recovery site350entry upon an actual disaster or disaster test.

The XML document containing the private cloud metadata may also contain information concerning physical machines which are not directly part of the virtual is infrastructure. For example, a database application may run on a non-virtualized server. This non-virtualized machine may be necessary for the proper operation of applications running within virtual machines but it is not necessary for the operation of the virtual machines themselves more generally. As such, the Host to Recovery target mapping process shown inFIG. 7may also accommodate physical-machine to physical-machine mapping. Continuing the example, the OS for the physical database server is thus installed via the same automated process that handles the management server and the host machines, but that process is instead carried out on a recovery target610that is a dedicated physical machine.

Automated systems for bare-metal operating system installation715often make use of a technique by which the target machine boots from the network, rather than internal storage. The technique used by this system is no different. This presents a challenge when recreating a network which existed in the protected site. In most cases, the network used for deployment on the recovery site350will not be compatible with the network to be recreated. If this is not addressed, the import tool600that recovers the hosts610-E,610-F,610-G (which is run following bare metal OS installation715), would be unable to contact the recovered management server610-D. This problem can be solved by configuring an additional network interface on the recovered management server610-D, which was not configure on the original management server102running in the protected site300. The result is that recovered management server610-D has one network interface on the network used for automation and OS installation, and one or more other network interfaces on the network, as specified in the metadata. The recovered management server610-D is therefore connected to two networks simultaneously, using different network interfaces.

At this point, the recovery target machines610have the required OS or hypervisor software installed, but no storage is connected and the machines are not aware of each other. Further the management recovery machine610-D has no information about virtual machines or virtual network configuration. The diagram ofFIG. 8shows the next step: connecting storage. There are no lines connecting the management server610-D to the hosts610-E,610-F,610-G, because this logical is connection has not yet been made.

During the protected host to recovery host mapping process, it is also possible for an administrative user to choose which virtual machines110will be recovered. This is similar, but simpler, to mapping hosts. It only requires a deploy flag to be set to true or false depending upon if the host is to be recovered, no mapping is required. An example representation:

Finally, a metadata import tool600imports metadata710from the repository into the management server610-D. The import tool600links the hosts610-E,610-F,610-G with the management server610-D, makes the management server610-D aware of the storage, creates any required virtual networks within the hosts and registers virtual machines110.

Metadata import600is a complement to the metadata capture410process. Data must be imported in the correct order or the import600will fail. For example, the protected site300includes other data processing resources111typically needed to implement a functioning Data Center, host clusters, and the like (seeFIG. 4). Those other data processing resources111, when instantiated as recovered resources611may need to be imported and brought on line prior to the hosts104being recovered. Recovered hosts610must also typically be imported prior to certain other resources611such as guests, virtual switches, and resource pools. Other storage resources611must also typically be configured after hosts but prior to guests. It is also important to avoid attempting to add resources which have already been imported. Virtual switch resources611, for example, may not be imported twice.

The metadata import tool600thus contains and/or determines the necessary import order taking these considerations into account. For example, if the metadata import tool600encounters a pre-existing network item, such as a switch which exists in a default configuration, the metadata import tool600switches to a mode where it augments the recovery of that switch rather than attempting to add a duplicate.

Machines used as hosts104in the protected site300often contain many network interfaces. But during the early phases of the recovery process, there may be only a single network interface connected on the recovery site350. More typically a recovered host610-E will have one or more network interfaces configured to allow remote management of that host610-E. If the network interface to the recovered management server610-D, as specified in the metadata, is not physically connected at the time of metadata capture410the recovered management server610-D will have lost a required management connection to that recovered host610-E. The import tool600can detect this situation, prior to attempting the metadata import, and re-arrange the recovered network interfaces in a manner compatible with the original configuration, but ensuring that the recovered management interface is matched to a physically connected port on the recovered host610-E.

Referring now toFIGS. 8 and 9, a process for recovering a private cloud virtual infrastructure can therefore proceed as follows.

At an initial time of recovery there are not yet any target machines assigned to replace the hosts or the management server. Thus the recovery process much first go through a mapping where the metadata is used to map the hosts (as specified by the metadata) to replacement machines available on the recovery site. The replacement machines may be maintained as a resource pool, and only allocated to recovery of a particular site on demand.

Once this physical machine mapping finishes, the recovery process can then perform an initial bare metal provisioning of each such recovery target. This bare metal provisioning can install software necessary for the recovery target hardware to become virtual machine hosts and/or the management server. The storage arrays at this point are not yet connected and all we have done is to prepare an environment in which to recover the virtual machines.

Once the bare metal provisioning is complete, storage for the virtual machines as available via the replicated storage portions (provided for example via replication services inherent in the storage area network (SAN) itself) are connections are connected to the respective recovery target machines. At this point the respective recovery target machines still do not have information about the virtual machines in which they are expected to host and/or any connection to the management server.

Only at this point does the process consider the metadata further via a metadata import process, and first inform the management server about the hosts that are part of its cluster. The management server can then connect to the hosts, and thus allow for recovery of the virtual machines.

Partial recovery is possible as part of this process. In particular, it may not be desirable or necessary to recovery the entire virtual infrastructure, and the user can specify at the time of recovery, which particular virtual machines to be recovered.

It should be understood that the example embodiments described above may be implemented in many different ways. In some instances, the various “data processors” described herein may each be implemented by a physical or virtual general purpose computer having a central processor, memory, disk or other mass storage, communication interface(s), input/output (I/O) device(s), and other peripherals. The general purpose computer is transformed into the processors and executes the processes described above, for example, by loading software instructions into the processor, and then causing execution of the instructions to carry out the functions described.

As is known in the art, such a computer may contain a system bus, where a bus is a set of hardware lines used for data transfer among the components of a computer or processing system. The bus or busses are essentially shared conduit(s) that connect different elements of the computer system (e.g., processor, disk storage, memory, is input/output ports, network ports, etc.) that enables the transfer of information between the elements. One or more central processor units are attached to the system bus and provide for the execution of computer instructions. Also attached to system bus are typically I/O device interfaces for connecting various input and output devices (e.g., keyboard, mouse, displays, printers, speakers, etc.) to the computer. Network interface(s) allow the computer to connect to various other devices attached to a network. Memory provides volatile storage for computer software instructions and data used to implement an embodiment. Disk or other mass storage provides non-volatile storage for computer software instructions and data used to implement, for example, the various procedures described herein.

Embodiments may therefore typically be implemented in hardware, firmware, software, or any combination thereof.

The computers that execute the processes described above may be deployed in a cloud computing arrangement that makes available one or more physical and/or virtual data processing machines via a convenient, on-demand network access model to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. Such cloud computing deployments are relevant and typically preferred as they allow multiple users to access computing resources as part of a shared marketplace. By aggregating demand from multiple users in central locations, cloud computing environments can be built in data centers that use the best and newest technology, located in the sustainable and/or centralized locations and designed to achieve the greatest per-unit efficiency possible.

In certain embodiments, the procedures, devices, and processes described herein are a computer program product, including a computer readable medium (e.g., a removable storage medium such as one or more DVD-ROM's, CD-ROM's, diskettes, tapes, etc.) that provides at least a portion of the software instructions for the system. Such a computer program product can be installed by any suitable software installation procedure, as is well known in the art. In another embodiment, at least a portion of the is software instructions may also be downloaded over a cable, communication and/or wireless connection.

Embodiments may also be implemented as instructions stored on a non-transient machine-readable medium, which may be read and executed by one or more procedures. A non-transient machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a non-transient machine-readable medium may include read only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; and others.

Furthermore, firmware, software, routines, or instructions may be described herein as performing certain actions and/or functions. However, it should be appreciated that such descriptions contained herein are merely for convenience and that such actions in fact result from computing devices, processors, controllers, or other devices executing the firmware, software, routines, instructions, etc.

It also should be understood that the block and network diagrams may include more or fewer elements, be arranged differently, or be represented differently. But it further should be understood that certain implementations may dictate the block and network diagrams and the number of block and network diagrams illustrating the execution of the embodiments be implemented in a particular way.

Accordingly, further embodiments may also be implemented in a variety of computer architectures, physical, virtual, cloud computers, and/or some combination thereof, and thus the computer systems described herein are intended for purposes of illustration only and not as a limitation of the embodiments.