Synchronizing snapshot volumes across hosts

Prior to overwriting a block of data in a first volume of data on a primary host, the block of data is written to a first snapshot of the first volume. Subsequently, the first snapshot can be synchronized with a snapshot of a second volume of data on a secondary host, where the second volume is a replica of the first volume. To synchronize the snapshots, only a portion of the first snapshot (e.g., the block of data that was written to the first snapshot) is sent to the secondary host.

BACKGROUND

In virtual machine (e.g., virtual server) environments, a “golden image” is essentially a template that serves as the source of applications, databases, file systems, and other information with which virtual machines are provisioned. Virtual machines can be provisioned by cloning or replicating the golden image. For example, boot volumes for virtual machines can be provisioned by cloning the boot volume of the golden image.

In order to make each virtual machine ready for disaster recovery, the virtual disks of each virtual machine are replicated to remote sites over a wide area network (WAN). After they are created, the remote copies need to be periodically synchronized with the source copy—that is, after they are created, the remote copies need to be kept up-to-date if and when the source copy is modified.

Snapshots are also used to record and preserve the state of a storage device (e.g., a virtual disk) at any given moment. For disaster recovery, the snapshot volumes at the local and remote sites also need to be synchronized. However, standard synchronization mechanisms can be problematic when used for synchronizing snapshots. For example, if standard synchronization mechanisms are applied to snapshots, the copy of a virtual disk at the remote site may end up being larger than the corresponding source (local) version of the virtual disk. Also, synchronizing snapshots using standard synchronization mechanisms can take a relatively long time to complete and/or can consume a relatively large amount of network bandwidth.

SUMMARY

In one embodiment, prior to overwriting a block of data in a first volume of data on a primary host, the block of data is written to a first snapshot of the first volume. Subsequently, the first snapshot can be synchronized with a snapshot of a second volume of data on a secondary host, where the second volume is a replica of the first volume. To synchronize the snapshots, only a portion of the first snapshot (e.g., the block of data that was written to the first snapshot) is sent to the secondary host. Accordingly, snapshots can be efficiently synchronized across hosts.

In another embodiment, a first snapshot of a first volume of data is accessed. The first snapshot includes valid regions and invalid regions, where a region of the first snapshot is characterized as a valid region if the region includes data written from the first volume to the first snapshot, and otherwise the region is characterized as an invalid region. Subsequently, the first snapshot can be synchronized with a snapshot of a second volume of data on a secondary host, where the second volume is a replica of the first volume. To synchronize the snapshots, the valid regions but not the invalid regions are written to the second snapshot.

In yet another embodiment, a virtual disk includes a first volume of data stored in an array of physical disks on a primary host. A volume manager can: replicate the first volume of data to create a second volume of data on a secondary host; create a first snapshot of the first volume; write blocks of data to the first snapshot prior to overwriting those blocks of data with new data; and synchronize the first snapshot and a second snapshot of the second volume of data by sending only a portion of the first snapshot (e.g., the blocks of data that were written to the first snapshot) to the secondary host.

These and other objects and advantages of the various embodiments of the present disclosure will be recognized by those of ordinary skill in the art after reading the following detailed description of the embodiments that are illustrated in the various drawing figures.

DETAILED DESCRIPTION

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present disclosure, discussions utilizing terms such as “accessing,” “writing,” “overwriting,” “synchronizing,” “sending,” “setting,” “pointing,” “replicating,” “creating,” “storing,” “updating,” “modifying,” “producing,” or the like, refer to actions and processes (e.g., flowchart900ofFIG. 9) of a computer system or similar electronic computing device or processor (e.g., system210ofFIG. 2). The computer system or similar electronic computing device manipulates and transforms data represented as physical (electronic) quantities within the computer system memories, registers or other such information storage, transmission or display devices.

Embodiments described herein may be discussed in the general context of computer-executable instructions residing on some form of computer-readable storage medium, such as program modules, executed by one or more computers or other devices. By way of example, and not limitation, computer-readable storage media may comprise computer storage media and communication media. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or distributed as desired in various embodiments.

FIG. 1is a block diagram depicting a network architecture100in which nodes110and120are coupled to a network150, which may be a WAN such as the Internet. Each of the nodes110and120can be implemented using computer system210ofFIG. 2. The node110may be referred to as a source location, and the node120may be referred to as a remote location. The source node and remote node may also be referred to as a primary host and a secondary host, respectively. There may be multiple secondary hosts associated with each primary host.

The nodes110and120can communicate via the network150but otherwise are physically independent of one another. In other words, the two nodes are “single failure proof,” such that a single failure or event would not be expected to disable both nodes, thus allowing the node120to serve as a backup for the node110(or vice versa). As will be seen by the discussion herein, a consistent copy of data on the source node110can be maintained at the remote node120. The copy of the data at the remote location can be used for disaster recovery (DR) or for other purposes.

FIG. 2depicts a block diagram of a computer system210suitable for implementing the present disclosure. The computer system210includes a bus212which interconnects major subsystems of the computer system. These subsystems include a central processor214; a system memory217; an input/output controller218; an external audio device, such as a speaker system220via an audio output interface222; an external device, such as a display screen224via display adapter226; serial ports228and230; a keyboard232(interfaced with a keyboard controller233); a storage interface234; a floppy disk drive237operative to receive a floppy disk238; a host bus adapter (HBA) interface card235A operative to connect with a Fibre Channel network290; an HBA interface card235B operative to connect to a Small Computer System Interface (SCSI) bus239; and an optical disk drive240operative to receive an optical disk242. Also included are a mouse246(or other point-and-click device, coupled to bus212via serial port228); a modem247(coupled to bus212via serial port230); and a network interface248(coupled directly to bus212). The modem247, network interface248or some other method can be used to provide connectivity from each of the nodes110and120to the network150ofFIG. 1.

The bus212ofFIG. 2allows data communication between the central processor214and system memory217, which may include ROM or flash memory and RAM (not shown), as previously noted. The RAM is generally the main memory into which the operating system and application programs are loaded. The ROM or flash memory can contain, among other code, the Basic Input-Output System (BIOS) which controls basic hardware operation such as the interaction with peripheral components.

Applications resident within the computer system210are generally stored on and accessed via a computer-readable storage medium, such as a hard disk drive (e.g., the fixed disk244), an optical drive (e.g., the optical drive240), a floppy disk unit237, or other storage medium. The computer-readable storage medium may be implemented as one or more virtual disks residing on an array of physical disks, as discussed in conjunction withFIG. 3. Applications can be in the form of electronic signals modulated in accordance with the application and data communication technology when accessed via network modem247or interface248.

Continuing with reference toFIG. 2, storage interface234, as with the other storage interfaces of computer system210, can connect to a standard computer-readable storage medium for storage and/or retrieval of information, such as a fixed disk drive244. The fixed disk drive244may be a part of the computer system210, or it may be separate and accessed through other interface systems. The modem247may provide a direct connection to a remote server via a telephone link or to the Internet via an internet service provider (ISP). The network interface248may provide a direct connection to a remote server via a direct network link to the Internet via a POP (point of presence). The network interface248may provide such a connection using wireless techniques, including digital cellular telephone connection, Cellular Digital Packet Data (CDPD) connection, digital satellite data connection or the like.

Many other devices or subsystems (not shown inFIG. 2) may be connected in a similar manner (e.g., document scanners, digital cameras and so on). Conversely, all of the devices shown inFIG. 2need not be present to practice the present disclosure. The devices and subsystems can be interconnected in different ways from that shown inFIG. 2.

The operation of a computer system such as that shown inFIG. 2is readily known in the art and is not discussed in detail in this application. Code to implement the present disclosure can be stored in computer-readable storage media such as one or more of the system memory217, fixed disk244, optical disk242, or floppy disk238. The operating system provided on the computer system210may be MS-DOS®, MS-WINDOWS®, OS/2®, UNIX®, Linux®, or another known operating system.

FIG. 3is a block diagram illustrating an embodiment of a volume management system300. The system300can be implemented as computer-executable components residing on a computer-readable storage medium. For example, the system300can be implemented as part of a virtual machine that is implemented using the system210ofFIG. 2.

With reference toFIG. 3, a volume manager310operates as a subsystem between an operating system320and one or more volumes305of data (blocks of data) that are stored in a data storage system (seeFIG. 2for examples). The data storage system may include physical storage devices such as an array of physical disks, LUNs (SCSI logical units), or other types of hardware that are used to store data. The volume manager310overcomes restrictions associated with the physical storage devices by providing a logical volume management layer that allows data to be spread across multiple physical disks within a disk array (not shown) in a manner that is transparent to the operating system320, in order to distribute and/or balance input/output (I/O) operations across the physical disks. Each of the volumes305can retrieve data from one or more of the physical storage devices, and the volumes305are accessed by file systems, databases, and other applications in the same way that physical storage devices are accessed. The operation of the volume manager310in the manner just described is well known.

FIG. 4is a block diagram illustrating an embodiment of a data replication process. The volumes and snapshots shown inFIG. 4can be implemented as computer-executable components residing on a computer-readable storage medium (e.g., in system210ofFIG. 2).

A first volume410of data resides on the primary host110(e.g., a first virtual machine) and includes one or more blocks of data including a block411(which may also be referred to herein as the first block of data). The volume410is analogous to the volume305ofFIG. 3.

In one embodiment, the first volume410ofFIG. 4is a “boot volume” and includes operating system files and supporting files, although this disclosure is not so limited. The first volume410can represent a “golden image” (e.g., a template that serves as the source of applications, databases, file systems, and other information with which virtual machines can be provisioned).

In process A and at time T0, the first volume410is replicated on the secondary host120(e.g., a second virtual machine) to create a second volume430of data including a block431(which is a replica of the block411). In this manner, a consistent copy of the first volume410is provided on a remote host for disaster recovery or for other purposes.

In process B, a snapshot420(also referred to herein as a first snapshot) of the first volume410is created. Process B may be performed before, after, or concurrently with process A; in general, process B is performed before new data is written to the first volume410.

In process C, a snapshot440(also referred to herein as a second snapshot) of the second volume430is created on the secondary host120. Thus, at the point in time in which the second volume430is replicated, the second snapshot440is a replica of the first snapshot420. The snapshots420and440may also be known as volume snapshots.

In one embodiment, the snapshots420and440are copy-on-write (COW) snapshots; however, the snapshots may be any kind of snapshot, including snapshots that do not include copy-on-write. If the snapshots are COW snapshots, then write operations to the first volume410result in a COW operation (see the examples ofFIGS. 5A and 5B).

In one embodiment, metadata or pointers425and445are included in, or associated with, the snapshots420and440, respectively. For example, the first snapshot420includes a region421that corresponds to the block411of the first volume410; at the time that the first snapshot420is created, the metadata425associated with the region421points to the block411. At the time that the second snapshot440is created, the metadata445is a reflection of the metadata425. That is, the metadata445is based on the metadata425, except that the metadata445points to the second volume430(e.g., the metadata associated with the region441points to the block431).

Continuing with reference toFIG. 4, in one embodiment, the first snapshot420includes or is associated with a backend storage element450that functions as a persistent store for the data resulting from writes to the original volume410or due to modifications to the first snapshot420. In a similar manner, the second snapshot440includes or is associated with a backend storage element460that functions as a persistent store for the contents of the second snapshot440. In the event of a write operation to the block411, for example, the existing block of data is copied from the first volume410to the storage element450before new data is written to that block. The storage element450can be shared by multiple volumes in the same disk group.

In general, a block of data is copied from the first volume410to the first snapshot420(e.g., to the storage element450) only if there is a write to that block of data. Thus, the data in the first volume410may change; however, the first snapshot420provides a stable and independent copy of the original data (where “original data” refers to the data in the volume410at time T0; original data may also be referred to herein as “valid data”) that can be used for disaster recovery or for other purposes. The first snapshot420requires less storage space than the first volume410because the snapshot420stores only the data that has changed and otherwise merely contains a pointer to the first volume410. As such, the first snapshot420may be referred to as a “space-optimized snapshot.”

With reference now toFIGS. 5A and 5B, an example of a write operation is described. In the example ofFIG. 5A, the first volume410includes a first block411of data D0and a second block412of data. Initially (at time T0), the first snapshot420includes or is associated with metadata425that includes pointers521and522that point to the blocks411and412, respectively, as previously described herein. At a point in time T1after time T0, an attempt to write data D1to the first block411is detected.

In the example ofFIG. 5B, the data D0in the first block411is copied to the first snapshot420(e.g., to the region421in the storage element450) before the new data D1is written to that block. After the original data D0is copied to the first snapshot420, the write can be committed and the new data D1can be written to the first block411. Thus, the first snapshot420contains the data D0originally stored in the first block411and the metadata425; the first block411in the first volume410stores the new data D1; and the second block412continues to store original data.

In general, at any point in time, original data will reside either entirely in the first volume410(if there have been no write operations), entirely in the first snapshot420(if every block in the first volume410has been written), or partially in the first volume410and partially in the first snapshot420. However, the lifetime of the snapshot420is relatively short compared to the lifetime of the first volume410—a new snapshot of the first volume410will likely be taken before every block in the first volume410has been written. Therefore, as pointed out above, the snapshot420generally will be smaller than the first volume410.

Regions of the first snapshot420that correspond to blocks in the first volume410that have been written with new data may be characterized as “valid” regions in which valid (original) data resides, while the other regions of the snapshot420may be characterized as “invalid” regions. In other words, the first snapshot420can include valid regions and invalid regions, where valid regions are those regions in the snapshot for which valid data exists in the snapshot420, while invalid regions in the snapshot are those for which the valid data exists in the first volume410. In the example ofFIG. 5B, the region421is characterized as a valid region, while the region422is characterized as an invalid region. In a snapshot, only the valid regions hold data, and therefore the first snapshot420is smaller than and will require less storage space than the first volume410.

In general, the metadata425(e.g., the pointers) point to the valid data. For a particular region (e.g., region422) of the first snapshot420, if valid data resides in the first volume410, then the pointer for that region points to the corresponding region (e.g., region412) of the first volume. If valid data resides in a particular region (e.g., region421) of the first snapshot420, then the pointer essentially points to that region.

In one embodiment, the metadata425associated with the first snapshot420is updated after the original data D0is copied to the first snapshot420. As noted above, a purpose of the snapshot420is to restore the original data (the data that was present at time T0) to the first volume410for disaster recovery. At some point in time T(N) after time T0, some blocks of original data for the first volume410may still reside in the first volume410while other blocks of original data may reside in the first snapshot420. At time T(N), the metadata425will still include pointers to those blocks of the first volume410that contain original (valid) data. Thus, for example, after the data D0in the first block411is copied to the first snapshot420, the metadata425can be updated so that the pointer521points to the block421; the metadata425will still contain the pointer522that points to the block412.

FIG. 6is an example of a data change map600that can be used to track which regions of the first snapshot420(FIGS. 5A and 5B) are valid regions and which regions of the snapshot are invalid regions. The data change map600can reside in memory of the system210(FIG. 2) and can be maintained by the volume manager310(FIG. 3). In the example ofFIG. 6, the data change map600is implemented as a bitmap; however, the data change map can be implemented in other ways. For example, the data change map600can be implemented as a series of bits that represent multiple states. In general, the data change map600can be implemented in any manner that describes whether a region is valid or invalid.

In the example ofFIG. 6, each bit in the bitmap that is set (e.g., has a value of one) represents a valid region of the snapshot420. In other words, each set bit represents a region of the first snapshot420(e.g., the storage element450) that holds data that is different from the corresponding block in the first volume410; each set bit represents a block or a region of blocks in the first volume410that has been written since the snapshot420was created. In the example ofFIGS. 5B and 6, the bit611is set (e.g., it has a value of one) to indicate that the region421is a valid region, and the bit612is clear (e.g., it has a value of zero) to indicate that the region422is an invalid region.

The data change map600can be used to supplement or instead of the metadata425ofFIGS. 5A and 5B. For example, instead of using pointers to point to the valid regions, the data change map600can be used to infer which regions are valid, and then data can be fetched on demand from the appropriate volume (that is, from either the first volume410or the first snapshot420, whichever is appropriate).

With reference now toFIG. 7, the first snapshot420(on the primary host110) and the second snapshot440(on the secondary host120) are “synchronized” at a point in time T2(after time T1). To synchronize the snapshots, only the valid regions of the first snapshot are copied to the second snapshot over the WAN150(FIG. 1). Because only the valid regions of the first snapshot420are sent/copied to the second snapshot440, the two snapshots will be of the same size.

As noted previously herein with reference toFIG. 4, when the second volume430and the second snapshot440are created (at time T0), the metadata445associated with the second snapshot440is based on the metadata425. When the first and second snapshots are updated at time T2, the metadata445can be updated as part of the snapshot synchronization process. Alternatively, the metadata445can be replaced with a copy of the updated metadata425sent from the primary host110along with the valid regions of the first snapshot420; however, the copy of the metadata425is revised as needed so that the new metadata445will point to the second volume430and/or the second snapshot440, as appropriate.

As discussed above, in one embodiment, the list of valid regions can be determined using the data change map600ofFIG. 6. After the first snapshot420and second snapshot440are synchronized so that the second snapshot is a replica of the first snapshot, the data change map600can be initialized (e.g., set bits can be cleared).

The first volume410and the first snapshot420can be quiesced while the snapshots420and440are synchronized; that is, writes to the first volume410and the first snapshot420can be temporarily suspended while the snapshots are synchronized. Because the synchronization will occur relatively quickly, suspending writes to the first volume410is not expected to have a significant impact on performance.

However, the snapshots420and440can be synchronized without suspending writes to the first volume410using a process referred to herein as “replay.” Using replay, the data change map600can be accessed again (after time T2) to identify any region of the first snapshot420that contains new data written to the snapshot420after time T2. As noted above, the data change map600is initialized after the first and second snapshots are synchronized. Thus, if the data change map600contains any values different from the initialized values (e.g., any set bits) when the data change map is accessed after time T2, that means that data has been copied to the first snapshot420since the synchronization at time T2. Consequently, the snapshot synchronization process may be repeated.

The discussion above is summarized inFIGS. 8A,8B, and8C, which depict an embodiment of a snapshot synchronization process. At time T0(FIG. 8A), the first volume410is replicated to create the second volume420. Also, a first snapshot420of the first volume410is created, and a second snapshot440of the second volume420is created. At this point, the first snapshot420only includes metadata (pointers)425that point to the first volume410, and the second snapshot440only includes metadata (pointers)445that point to the second volume430.

At time T1(FIG. 8B), a write operation is detected, in which a block of data containing data D0is targeted to be overwritten with new data. As described above in conjunction withFIGS. 5A and 5B, before the write operation is committed, the original data D0is copied to the first snapshot420. Also, the metadata425associated with the first snapshot420is updated (pointers that point to valid regions in the first snapshot are not shown).

At time T2(FIG. 8C), the first and second snapshots are synchronized as described above in conjunction withFIG. 7. Thus, the second snapshot440also now includes the original block of data D0. Also, the metadata445associated with the second snapshot440is updated (pointers that point to valid regions in the second snapshot are not shown).

While the examples above are described in the context of a single block of data and a single synchronization, the present disclosure is not so limited. The examples above can be readily extended to scenarios in which multiple blocks of data are copied from the first volume410to the first snapshot420between times T1and T2. Also, the examples above can be readily extended to scenarios in which the first and second snapshots are synchronized more than once. For example, at some time T3after time T2, the first snapshot420may be further modified, and at some time T4after time T3, the first and second snapshots may again be synchronized in the manner described herein. In general, any number of snapshots of the first volume can be synchronized with the snapshot on the secondary host.

FIG. 9is a flowchart900illustrating an embodiment of a computer-implemented method for managing data and synchronizing snapshots. Flowchart900can be implemented as computer-executable instructions residing on some form of computer-readable storage medium (e.g., in the system210ofFIG. 2). For example, the method of flowchart900can be implemented by the volume manager310ofFIG. 3.FIG. 9is described in conjunction withFIG. 4unless otherwise noted.

In block902, a first volume of data410on a primary host110is replicated to create a second volume of data430on a secondary host120. The first volume410can be, but is not limited to, a virtual disk, boot volume, or golden image, and is not restricted to a single physical disk.

In block904, a first snapshot420(e.g., a COW snapshot) of the first volume410is created. The snapshot420includes, or is associated with, metadata425that points to blocks of data in the first volume410.

In block906, before a block of data (e.g., block411) is overwritten, the block411is written (copied) to the first snapshot420. In one embodiment, the block411is written to a storage element450. In one embodiment, a data change map600(FIG. 6) is updated to indicate that a region of the snapshot420corresponding to the block411(e.g., the region421ofFIG. 5B) has been written with the data from that block. That is, the data change map600is updated to indicate that the region421is a valid region. Also, the metadata425associated with the first snapshot420can be updated.

The operations associated with block906may be repeated before the next step in the flowchart900(block908) is performed. In other words, more than one block of data in the first volume410may be copied to the first snapshot420before the first snapshot is synchronized with the second snapshot440.

In block908ofFIG. 9, with reference also toFIGS. 5A and 5B, the first snapshot420(as modified in block906) and the second snapshot440are synchronized by sending only a portion of the snapshot420(e.g., only the block411that was written to the first snapshot) to the secondary host120. More specifically, only the valid regions, and not the invalid regions, of the first snapshot420are sent to the secondary host120and written to the second snapshot440.

When the first and second snapshots are synchronized, the metadata445for the second volume430can be updated as part of the snapshot synchronization process. Alternatively, the metadata445can be replaced with a copy of the updated metadata425associated with the first volume410; however, the copy of the metadata425is revised as needed so that the new metadata445will point to the second volume430.

Consequently, the copy of the virtual disk at the remote site (e.g., the second volume430and second snapshot440on the secondary host120) is not larger than the corresponding original version of the virtual disk (e.g., the first volume410and first snapshot420on the primary host110). Also, the process of synchronizing snapshots can be accomplished more quickly and using less bandwidth than conventional techniques.

The operations associated with block908may be repeated. In other words, the first snapshot420may be synchronized with the second snapshot440at some time T(M); at time T(M+1), the first snapshot may be modified as described in conjunction with block906; at time T(M+2), the first and second snapshots may again be synchronized; and so on.

Furthermore, after the first and second snapshots are synchronized at some point in time, an entirely new snapshot of the current state of the volume410of data residing on the primary host at that point in time can be created. In other words, the flowchart900can return to block902, and the operations associated with the flowchart900can be repeated.

Embodiments according to the invention are thus described. While the present disclosure has been described in particular embodiments, it should be appreciated that the invention should not be construed as limited by such embodiments, but rather construed according to the below claims.