Implementing snapshot sets for consistency groups of storage volumes

Techniques for implementing snapshot sets for consistency groups of storage volumes in data storage systems. For each snapshot in a snapshot set of a consistency group, metadata is expanded to include a snapshot set identifier, and a consistency group identifier. The data structure of the consistency group is enhanced to include a snapshot set list, which includes the snapshot set identifier(s) of the consistency group, and a pointer to a storage volume list that contains names of storage volumes with snapshots in the snapshot set(s). By expanding the metadata of each snapshot in a snapshot set, and enhancing the data structure of each consistency group with snapshot set(s), relationships between snapshots, snapshot sets, consistency groups, and storage volumes can be readily maintained, allowing storage volumes to be added to and/or removed from consistency groups without having to delete snapshots of the storage volumes and/or snapshot sets of the consistency groups.

BACKGROUND

Conventional data storage systems create snapshots of storage volumes at certain points-in-time (PITs) for a variety of purposes, including, but not limited to, backing-up volumes of data, providing operational and/or disaster recovery, improving application availability, and reducing data management costs. Such snapshots of storage volumes are typically used to record states of the data storage systems, as well as provide point-in-time (PIT) views of consistency groups, which represent sets of storage volumes of applications stored on storage appliances within the data storage systems. By maintaining consistency among data stored on the sets of storage volumes of the consistency groups, such applications can successfully recover from failures (e.g., database errors, etc.) using consistent data for the storage volumes at one or more PITs of the data storage systems. Such consistent data for the storage volumes can also be used for data recovery in the event an accident, a disaster, or other catastrophic failure causes data at a production site to be lost.

SUMMARY

In conventional data storage systems, storage appliances can store snapshots of storage volumes at certain PITs of the data storage systems, as well as store sets of snapshots (also referred to herein as “snapshot set(s)”) of storage volumes within consistency groups at the respective PITs. However, maintaining storage volumes with one or more associated snapshots and/or consistency groups with one or more associated snapshot sets on such conventional data storage systems can restrict the data storage systems' ability to add storage volumes to the consistency groups, as well as remove storage volumes from the consistency groups. For example, in order to add a storage volume that has one or more associated snapshots to a consistency group, a conventional data storage system must typically delete all of the snapshots of the storage volume before adding the storage volume to the consistency group. Further, in order to add a storage volume to (or remove a storage volume from) a consistency group that has one or more associated snapshot sets, the conventional data storage system must typically delete all of the snapshot sets of the consistency group before adding the storage volume to (or removing the storage volume from) the consistency group. However, having to delete snapshots and/or snapshot sets before adding storage volumes to (or removing storage volumes from) consistency groups can reduce the amount of consistent data available to the data storage system, adversely affecting the data storage system's ability to achieve data recovery in the event of an accident, a disaster, or other catastrophic failure.

Techniques are disclosed herein for implementing snapshot sets for consistency groups of storage volumes in data storage systems. The disclosed techniques employ expanded metadata for snapshots of storage volumes stored on the data storage systems, as well as enhanced data structures for consistency groups of the storage volumes. For each snapshot of a storage volume in a snapshot set of a consistency group, the snapshot metadata is expanded to include an identifier of the snapshot set (also referred to herein as the “snapshot set identifier” or “SSID”), and an identifier of the consistency group (also referred to herein as the “consistency group identifier” or “CGID”). The data structure of the consistency group is enhanced to include a snapshot set list (also referred to herein as the “consistency group snapshot set list” or “CGSSL”), which includes the SSID of each snapshot set of the consistency group, and a pointer to a storage volume list that contains one or more names of storage volumes with associated snapshots in the snapshot set. By expanding the metadata of each snapshot of a storage volume in a snapshot set of a consistency group, and enhancing the data structure of each consistency group with one or more associated snapshot sets, relationships among snapshots, snapshot sets, consistency groups, and storage volumes can be more readily maintained in a data storage system, thereby allowing storage volumes to be added to and/or removed from consistency groups without first having to delete snapshots of the storage volumes and/or snapshot sets of the consistency groups.

In certain embodiments, a method of implementing snapshot sets for consistency groups of storage volumes in a data storage system includes forming a consistency group of storage volumes on the data storage system. The consistency group has a data structure. The method further includes creating, at a first point-in-time (PIT), a first set of snapshots of the storage volumes in the consistency group, and generating first snapshot metadata for each snapshot in the first set of snapshots. The first snapshot metadata includes an identifier of the first set of snapshots and an identifier of the consistency group. The method still further includes incorporating a snapshot set list including a first snapshot set node into the data structure of the consistency group. The first snapshot set node includes the identifier of the first set of snapshots and a first pointer to a first storage volume list of the storage volumes having the respective snapshots in the first set of snapshots. The method yet further includes performing at least one of (i) adding an additional storage volume to the consistency group, and (ii) removing a respective storage volume from among the storage volumes in the consistency group, without deleting any of the snapshots in the first set of snapshots in the consistency group.

In certain aspects, the method includes, having added the additional storage volume to the consistency group, creating, at a second PIT, a second set of snapshots of the storage volumes including the additional storage volume in the consistency group, and generating second snapshot metadata for each snapshot in the second set of snapshots. The second snapshot metadata includes an identifier of the second set of snapshots and the identifier of the consistency group. The method further includes adding a second snapshot set node to the snapshot set list. The second snapshot set node includes an identifier of the second set of snapshots and a second pointer to a second storage volume list of storage volumes having the respective snapshots in the second set of snapshots.

In certain further aspects, a snapshot of the additional storage volume is stored on the data storage system outside of the consistency group. The method further includes, having added the additional storage volume to the consistency group, allowing only read requests or delete requests pertaining to the snapshot of the additional storage volume to be acted upon by the data storage system.

In certain other aspects, the method includes, having removed the respective storage volume from among the storage volumes in the consistency group, allowing only read requests pertaining to the snapshot of the respective storage volume to be acted upon by the data storage system. The method further includes deleting the snapshot of the respective storage volume in the event that (i) the first set of snapshots is deleted, refreshed, or restored, (ii) the consistency group is removed from the data storage system, or (iii) the respective storage volume is removed from the data storage system. The respective storage volume is initially stored on the data storage system outside of the consistency group. The method still further includes subsequently removing the respective storage volume from the data storage system. Before removing the respective storage volume from the data storage system, the first snapshot metadata for the snapshot of the respective storage volume is accessed to obtain the identifier of the first set of snapshots and the identifier of the consistency group. The method yet further includes (i) having obtained the identifier of the first set of snapshots and the identifier of the consistency group, accessing the first snapshot set node of the snapshot set list incorporated into the data structure of the consistency group, (ii) accessing the first storage volume list using the first pointer included in the first snapshot set node, (iii) deleting a name of the respective storage volume from the first storage volume list, and (iv) deleting the snapshot of the respective storage volume from the consistency group, and deleting the first snapshot metadata for the snapshot of the respective storage volume.

In certain additional aspects, the method includes forming a second consistency group of storage volumes on the data storage system. The second consistency group has a second data structure. The method further includes creating, at a second PIT, a second set of snapshots of the storage volumes in the second consistency group, and generating second snapshot metadata for each snapshot in the second set of snapshots. The second snapshot metadata includes an identifier of the second set of snapshots and an identifier of the second consistency group. The method still further includes incorporating a second snapshot set list including a snapshot set node into the second data structure of the second consistency group. The snapshot set node includes the identifier of the second set of snapshots and a second pointer to a second storage volume list of the storage volumes having the respective snapshots in the second set of snapshots.

In certain further embodiments, a system for implementing snapshot sets for consistency groups of storage volumes in a data storage system includes at least one memory, and at least one processor operative to execute instructions out of the at least one memory to form a consistency group of storage volumes on the data storage system. The consistency group has a data structure. The processor is further operative to execute the instructions out of the memory to create, at a first point-in-time (PIT), a first set of snapshots of the storage volumes in the consistency group, and to generate first snapshot metadata for each snapshot in the first set of snapshots. The first snapshot metadata includes an identifier of the first set of snapshots and an identifier of the consistency group. The processor is still further operative to execute the instructions out of the memory to incorporate a snapshot set list including a first snapshot set node into the data structure of the consistency group. The first snapshot set node includes the identifier of the first set of snapshots and a first pointer to a first storage volume list of the storage volumes having the respective snapshots in the first set of snapshots. The processor is yet further operative to execute the instructions out of the memory to perform at least one of (i) adding an additional storage volume to the consistency group, and (ii) removing a respective storage volume from among the storage volumes in the consistency group, without deleting any of the snapshots in the first set of snapshots in the consistency group.

Other features, functions, and aspects of the invention will be evident from the Detailed Description that follows.

DETAILED DESCRIPTION

Techniques are disclosed herein for implementing snapshot sets for consistency groups of storage volumes in data storage systems. The disclosed techniques employ expanded metadata for snapshots of storage volumes stored on the data storage systems, as well as enhanced data structures for consistency groups of the storage volumes. For each snapshot of a storage volume in a snapshot set of a consistency group, the snapshot metadata is expanded to include an identifier of the snapshot set (also referred to herein as the “snapshot set identifier” or “SSID”), and an identifier of the consistency group (also referred to herein as the “consistency group identifier” or “CGID”). The data structure of the consistency group is enhanced to include a snapshot set list (also referred to herein as the “consistency group snapshot set list” or “CGSSL”), which includes the SSID of each snapshot set of the consistency group, and a pointer to a storage volume list that contains one or more names of storage volumes with associated snapshots in the snapshot set. By expanding the metadata of each snapshot of a storage volume in a snapshot set of a consistency group, and enhancing the data structure of each consistency group with one or more associated snapshot sets, relationships among snapshots, snapshot sets, consistency groups, and storage volumes can be more readily maintained in a data storage system, thereby allowing storage volumes to be added to and/or removed from consistency groups without first having to delete snapshots of the storage volumes and/or snapshot sets of the consistency groups.

FIG. 1depicts an illustrative embodiment of a data storage environment100, in which an exemplary data storage system104configured to implement snapshot sets for consistency groups of storage volumes can be employed. As shown inFIG. 1, the data storage environment100includes a plurality of host computers102.1,102.2, . . . ,102.ncommunicably coupled to the data storage system104by a communications medium103that includes a network110. For example, each of the plurality of host computers102.1,102.2, . . . ,102.ncan be configured as a web server computer, a file server computer, an email server computer, an enterprise server computer, a database server, and/or any other suitable computer or computerized device, for providing input/output (IO) requests (e.g., small computer system interface (SCSI) commands) to the data storage system104over the communications medium103. Such IO requests (e.g., write requests, read requests, delete requests) provided by the plurality of host computers102.1,102.2, . . . ,102.ncan direct the data storage system104to store and/or retrieve blocks of data to/from storage volumes (e.g., logical unit numbers (LUNs), virtual volumes (VVOLs)) on behalf of the respective host computers102.1,102.2, . . . ,102.n.

The communications medium103can be configured to interconnect the plurality of host computers102.1,102.2, . . . ,102.nwith the data storage system104to enable them to communicate and exchange electronic signals. As shown inFIG. 1, at least a portion of the communications medium103is illustrated as a “cloud” to indicate that the communications medium103can have a variety of different topologies including, but not limited to, backbone, hub-and-spoke, loop, irregular, or any suitable combination thereof. The communications medium103can also include, but is not limited to, copper-based data communications devices and cabling, fiber optic devices and cabling, wireless devices, or any suitable combination thereof. In addition, the communications medium103can be configured to support storage area network (SAN)-based communications, local area network (LAN)-based communications, cellular communications, wide area network (WAN)-based communications, distributed infrastructure communications, and/or any other suitable communications.

As shown inFIG. 1, the data storage system104can include at least one storage appliance106and a computerized snapshot set management system108, each of which can be coupled to an Ethernet sub-network112or any other suitable network and/or sub-network. The storage appliance106can include storage processing circuitry114, at least one memory116, and at least one storage device array118. The storage processing circuitry114can include one or more physical storage processors or engines, data movers, director boards, blades, IO modules, storage drive controllers, switches, and/or any other suitable computer hardware or combination thereof. The storage processing circuitry114in association with the memory116can be configured to process IO requests from one or more of the host computers102.1,102.2, . . . ,102.n, and store host data in a redundant array of independent disk (RAID) environment implemented by the storage device array118. In certain embodiments, the storage processing circuitry114and the memory116can be configured to process such IO requests, and store the host data in a cloud-based storage environment or any other suitable data storage environment. The memory116can include persistent memory storage (e.g., flash memory, magnetic memory) and/or non-persistent cache memory storage (e.g., dynamic random access memory (DRAM), static random access memory (SRAM)), and accommodate a variety of software constructs including, but not limited to, operating system code and data, data storage system code and data, and snapshot set implementation code, data, and metadata. The storage device array118can store a plurality of storage volumes128.1, . . . ,128.p,130.1, . . . ,130.q, each of which can be implemented as a LUN, a VVOL, or any other suitable storage unit or volume.

As further shown inFIG. 1, the computerized snapshot set management system108can include a consistency group controller120, an object handler122, a snapshot manager124, and a consistency group snapshot set list manager126, each of which can be implemented on the computerized snapshot set management system108using any suitable computer or processing hardware, software, or combination thereof. The consistency group controller120can be configured to manage operations of the object handler122, the snapshot manager124, and/or the consistency group snapshot set list manager126for managing snapshots of storage volumes, managing snapshot sets of storage volumes within consistency groups, as well as providing an application programming interface (API) for communicating with the storage appliance106. The object handler122can be configured, under the direction of the consistency group controller120, to create objects (e.g., LUNs, VVOLs, snapshots, snapshot sets), to modify objects, to delete objects, to add objects to consistency groups, to remove objects from consistency groups, and/or to obtain properties (e.g., identifiers, names) of objects stored on the storage device array118of the storage appliance106. The snapshot manager124can be configured, under the direction of the consistency group controller120, to manage creation, modification, and/or deletion of expanded metadata for snapshots of storage volumes (e.g., LUNs, VVOLs) created by the object handler122. The consistency group snapshot set list manager126can be configured, under the direction of the consistency goup controller120, to manage one or more snapshot set lists and storage volume lists for each consistency group of storage volumes stored on the storage device array118. The storage device array118can include one or more consistency groups122.1, . . . ,122.m, such as the consistency group122.1for the plurality of storage volumes128.1, . . . ,128.pand the consistency group122.mfor the plurality of storage volumes130.1, . . . ,130.q.

As employed herein, the term “snapshot” refers to a point-in-time (PIT) replication of a storage volume (e.g., LUN, VVOL) or other data object, along with its associated snapshot metadata. Such a snapshot can replicate a full copy of data from a storage volume (e.g., a base LUN), or a space-efficient copy by storing differences between a current version of the data and a prior version of the data at a PIT when a snapshot of the storage volume was last taken. Such space-efficient copies can be used to replicate snapshots on remote data storage systems. As employed herein, the term “snapshot set” refers to one or more snapshots of storage volumes in a consistency group taken at a certain PIT. As employed herein, the term “consistency group” refers to a set of storage volumes stored on a storage appliance, for which consistency among data stored on the storage volumes is desired to be maintained.

FIG. 2depicts an illustrative embodiment of a storage device array200of a storage appliance included in a conventional data storage system, which has a restricted ability to add storage volumes to (and remove storage volumes from) consistency groups. As shown inFIG. 2, the storage device array200is configured to store a plurality of storage volumes (e.g., LUNs), namely, a LUN204, a LUN206, a LUN208, and a LUN210. The LUN204has an associated LUN snapshot204(1) taken at a first PIT. As further shown inFIG. 2, the storage device array200maintains the LUN208and the LUN210provisioned in a consistency group202, which includes a plurality of snapshot sets, namely, (i) a snapshot set212containing a LUN snapshot208(1) and a LUN snapshot210(1) of the LUN208and the LUN210, respectively, taken at the first PIT, and (ii) a snapshot set214containing a LUN snapshot208(2) and a LUN snapshot210(2) of the LUN208and the LUN210, respectively, taken at a second PIT.

The conventional data storage system including the storage device array200ofFIG. 2has drawbacks, however, in that its ability to add one or more storage volumes (e.g., LUNs) to the consistency group202, as well as remove one or more storage volumes (e.g., LUNs) from the consistency group202, is restricted. For example, in order to add the LUN204that has the associated LUN snapshot204(1) to the consistency group202, the conventional data storage system must typically delete the LUN snapshot204(1) from the storage device array200before adding the LUN204to the consistency group202. This restricted ability of the conventional data storage system to add the LUN204to the consistency group202is illustrated by an “X” on a data path corresponding to reference numeral216. Further, in order to add the LUN206to the consistency group202that has the snapshot set212and the snapshot set214, the conventional data storage system must typically delete the snapshot set212and the snapshot set214from the consistency group202before adding the LUN206to the consistency group202. This restricted ability of the conventional data storage system to add the LUN206to the consistency group202is illustrated by an “X” on a data path corresponding to reference numeral220. Likewise, in order to remove the LUN210from the consistency group202that has the snapshot set212and the snapshot set214, the conventional data storage system must typically delete the snapshot set212and the snapshot set214from the consistency group202before removing the LUN210from the consistency group202. This further restricted ability of the conventional data storage system to remove the LUN210from the consistency group202is illustrated by an “X” on a data path corresponding to reference numeral218. However, having to delete snapshots and/or snapshot sets before adding one or more LUNs to (or removing one or more LUNs from) the consistency group202can reduce the amount of consistent data available to the conventional data storage system, adversely affecting the data storage system's ability to achieve data recovery in the event of an accident, a disaster, or other catastrophic failure.

During operation of the data storage system104(seeFIG. 1), the computerized snapshot set management system108can be used to avoid at least some of the restrictions placed on conventional data storage systems while adding one or more storage volumes to (or removing one or more storage volumes from) consistency groups, by employing expanded metadata for snapshots of storage volumes in snapshot sets of the consistency groups, and enhanced data structures for the consistency groups that include the storage volumes and the snapshot sets. For each snapshot of a storage volume in a snapshot set of a consistency group, the computerized snapshot set management system108expands the snapshot metadata to include an SSID of the snapshot set, and a CGID of the consistency group containing the snapshot set. Further, the computerized snapshot set management system108enhances the data structure of the consistency group to include a snapshot set list (CGSSL), which includes the SSID of each snapshot set of the consistency group, and, for each SSID, a pointer to a storage volume list that contains one or more names of storage volumes with associated snapshots in the snapshot set. By expanding the metadata of each snapshot of a storage volume in a snapshot set of a consistency group, and enhancing the data structure of each consistency group with one or more associated snapshot sets, relationships among snapshots, snapshot sets, consistency groups, and storage volumes can be more readily maintained in the data storage system104, thereby allowing storage volumes to be added to and/or removed from consistency groups without first having to delete snapshots of the storage volumes and/or snapshot sets of the consistency groups.

FIG. 3ais a diagram of exemplary expanded metadata302for a snapshot (also referred to herein as the “expanded snapshot metadata”) of a storage volume in a snapshot set of a consistency group stored on the data storage system104ofFIG. 1. As shown inFIG. 3a, the expanded snapshot metadata302includes an M+N bit header consisting of an M-bit SSID314and an N-bit CGID316, in which “M” and “N” can be equal to thirty-two (32) or any other suitable value or different values. The snapshot metadata318can include (i) a first timestamp indicating the time when IO requests provided by one or more of the host computers102.1,102.2, . . . ,102.nwere paused at the storage appliance106where the snapshot was taken, (ii) a second timestamp indicating the time when the snapshot was taken at the storage appliance106, and/or any other suitable snapshot metadata.

FIG. 3bis a diagram of an exemplary enhanced data structure304for a consistency group (also referred to herein as the “enhanced consistency group data structure”) with one or more associated snapshot sets stored on the data storage system104ofFIG. 1. As shown inFIG. 3b, the enhanced consistency group data structure304includes a consistency group (CG) data structure320, as well as additional entries pertaining to a CGSSL308, which includes a plurality of consistency group snapshot set nodes310.1, . . . ,310.Q(also referred to herein as the “CGSSN(s)”). The CGSSL308includes a CGSSN310.1, and so on up to a CGSSN310.Q, in which “Q” can be any suitable positive integer value. Each of the CGSSN310.1, . . . , CGSSN310.Qincludes an SSID of a snapshot set of the consistency group, and an address that points to a storage volume list containing one or more names of storage volumes having snapshots in the snapshot set. For example, the CGSSN310.1includes an M-bit SSID322.1of a first snapshot set of the consistency group, and a P-bit pointer324.1to a first storage volume list that contains one or more names of storage volumes with associated snapshots in the first snapshot set; and, the CGSSN310.Q similarly includes an M-bit SSID322.Q of a Qthsnapshot set of the consistency group, and a P-bit pointer324.Q to a Qthstorage volume list that contains one or more names of storage volumes with associated snapshots in the Qthsnapshot set, in which “P” can be equal to sixty-four (64) or any other suitable value. The CG data structure320can include entries pertaining to (i) a first set of timestamps indicating the times when IO requests provided by one or more of the host computers102.1,102.2, . . . ,102.nwere paused at the storage appliance106where the first through Qthsnapshot sets were taken, (ii) a second set of timestamps indicating the times when the first through Qthsnapshot sets were taken at the storage appliance106, and/or any other suitable CG data structure entries.

FIG. 3cis a diagram of “Q” storage volume lists306pointed to by the P-bit pointers324.1, . . . ,324.Q, respectively. As shown inFIG. 3c, the storage volume lists306include a first storage volume list pointed to by the P-bit pointer324.1, and so on up to a Qthstorage volume list pointed to by the P-bit pointer324.Q. The first storage volume list pointed to by the P-bit pointer324.1includes a first plurality of names of storage volumes (e.g., WWN LUN326.1, . . . , WWN LUN326.X) with associated snapshots in the first snapshot set identified by the M-bit SSID322.1, and so on up to a Qthplurality of names of storage volumes (e.g., WWN LUN328.1, . . . , WWN LUN328.Y) with associated snapshots in the Qthsnapshot set identified by the M-bit SSID322.Q, in which “X” and “Y” can be any suitable positive integer value(s). For example, the names of the storage volumes in the storage volume lists306can correspond to endpoint addresses that include worldwide names (WWNs), each of which is a unique identifier (e.g., a node name, a port name) consisting of an 8-byte number, or any other suitable storage volume name or identifier.

The operation of the data storage system104ofFIG. 1for implementing snapshot sets for consistency groups of storage volumes will be further understood with reference to the following illustrative example, as well asFIGS. 4a, 4b, and 4c.FIG. 4adepicts a storage device array400of a storage appliance within a data storage system (such as the data storage system104) at a first PIT (also referred to herein as the “time T1”). At the time T1, the storage device array400is configured to store a plurality of storage volumes, namely, a LUN414, a LUN416, a LUN418, and a LUN420, which has an associated LUN snapshot420(1) taken at the time T1. The LUN414and the LUN416are provisioned in a consistency group408, and the LUN418is provisioned in a consistency group410. The consistency group408includes a snapshot set430, which contains a LUN snapshot414(1) and a LUN snapshot416(1) of the LUN414and the LUN416, respectively, each taken at the time T1. It is noted that the LUN420and its associated LUN snapshot420(1) are not included in either of the consistency groups408and410, or any other consistency group within the storage device array400.

FIG. 4a(continued) depicts expanded snapshot metadata414.1and expanded snapshot metadata416.1for the LUN snapshot414(1) and the LUN snapshot416(1), respectively, in the snapshot set430, and an enhanced consistency group data structure408afor the consistency group408with the associated snapshot set430. The expanded snapshot metadata414.1for the LUN snapshot414(1) includes an SSID430.1for the snapshot set430, a CGID408.1for the consistency group408, and other snapshot metadata418.1. The expanded snapshot metadata416.1for the LUN snapshot416(1) includes the SSID430.1for the snapshot set430, the CGID408.1for the consistency group408, and other snapshot metadata418.2. The enhanced consistency group data structure408afor the consistency group408includes a CG data structure419.1and a CGSSL440.1. The CGSSL440.1includes a single node, i.e., a CGSSN450.1, containing the SSID430.1for the snapshot set430, and a pointer424.1to a storage volume list406.1of names of the LUNs414,416having the associated LUN snapshots414(1),416(1), respectively, in the snapshot set430. As shown inFIG. 4a(continued), the storage volume list406.1can include a worldwide name (WWN) of the LUN414, and a worldwide name (WWN) of the LUN416. In one embodiment, expanded snapshot metadata420.1for the LUN snapshot420(1) (which is a standalone LUN snapshot that is not included in a snapshot set of a consistency group) can have a format like the expanded snapshot metadata414.1and the expanded snapshot metadata416.1, with the exception that each of an SSID and a CGID of the expanded snapshot metadata420.1has a default value of 0xFFFF or any other suitable default value. As shown inFIG. 4a(continued), the expanded snapshot metadata420.1for the LUN snapshot420(1) includes the SSID having the default value of 0xFFFF, the CGID having the same default value of 0xFFFF, and snapshot metadata418.3.

FIG. 4bdepicts the storage device array400of the storage appliance within the data storage system (such as the data storage system104) at a second PIT (also referred to herein as the “time T2”). At the time T2, the data storage system operates to add the LUN420to the consistency group408, while maintaining the LUN snapshot420(1) outside of the consistency groups408,410. The data storage system further operates to remove the LUN414from the consistency group408, while maintaining the LUN snapshot414(1) in the snapshot set430of the consistency group408. The data storage system also maintains the LUN418within the consistency group410. In addition, the data storage system operates to create a snapshot set432containing a LUN snapshot416(2) and a LUN snapshot420(2) of the LUN416and the LUN420, respectively, each taken at the time T2. The data storage system further operates to take a LUN snapshot414(2) of the LUN414at the time T2. Like the LUN snapshot420(1), the LUN snapshot414(2) is maintained outside of the consistency groups408,410.

FIG. 4b(continued) depicts the expanded snapshot metadata414.1for the LUN snapshot414(1), the expanded snapshot metadata416.1for the LUN snapshot416(1), and the expanded snapshot metadata420.1for the LUN snapshot420(1), as well as expanded snapshot metadata416.2and expanded snapshot metadata420.2for the LUN snapshot416(2) and the LUN snapshot420(2), respectively, in the snapshot set432. The expanded snapshot metadata416.2for the LUN snapshot416(2) includes an SSID432.1for the snapshot set432, the CGID408.1for the consistency group408, and other snapshot metadata418.5. The expanded snapshot metadata420.2for the LUN snapshot420(2) includes the SSID432.1for the snapshot set432, the CGID408.1for the consistency group408, and other snapshot metadata418.6.

FIG. 4b(continued) further depicts an enhanced consistency group data structure408bfor the consistency group408that includes a CG data structure419.2and a CGSSL440.2. The CGSSL440.2includes two (2) nodes, namely, the CGS SN450.1and a CGSSN450.2. The CGSSN450.1contains the SSID430.1for the snapshot set430and the pointer424.1to the storage volume list406.1. The CGSSN450.2contains the SSID432.1for the snapshot set432and a pointer424.2to a storage volume list406.2of names of the LUNs416,420that have the associated LUN snapshots416(2),420(2), respectively, in the snapshot set432. As shown inFIG. 4b(continued), the storage volume list406.2can include the worldwide name (WWN) of the LUN416, and a worldwide name (WWN) of the LUN420. In one embodiment, expanded snapshot metadata414.2for the LUN snapshot414(2) (which is a standalone LUN snapshot that is not included in a snapshot set of a consistency group) can have a format like the expanded snapshot metadata414.1,416.1,416.2,420.2, with the exception that each of an SSID and a CGID of the expanded snapshot metadata414.2has a default value of 0xFFFF or any other suitable default value. As shown inFIG. 4b(continued), the expanded snapshot metadata414.2for the LUN snapshot414(2) includes the SSID having the default value of 0xFFFF, the CGID having the same default value of 0xFFFF, and snapshot metadata418.4.

In this example, the data storage system operates to add the LUN420to the consistency group408, while maintaining the LUN snapshot420(1) outside of the consistency group408and the snapshot set430within the consistency group408. Unlike conventional data storage systems, the LUN420can be added to the consistency group408without having to delete the LUN snapshot420(1), and/or to delete the LUN snapshot414(1) and the LUN snapshot416(1) in the snapshot set430. This is because the data storage system provides (i) the expanded snapshot metadata414.1for the LUN snapshot414(1), (ii) the expanded snapshot metadata416.1for the LUN snapshot416(1), (iii) the expanded snapshot metadata420.1for the LUN snapshot420(1), and (iv) the enhanced consistency group data structure408bfor the consistency group408, which together allow relationships among the LUN snapshots414(1),416(1),420(1), the snapshot set430, the consistency group408, and the LUNs414,416,420to be maintained in the storage device array400while the LUN420is being added to the consistency group408, as well as after the LUN420has been added to the consistency group408. It is noted that, once the LUN420has been added to the consistency group408, only read requests and/or delete requests pertaining to the LUN snapshot420(1) will be acted upon by the data storage system.

Moreover, the data storage system operates to remove the LUN414from the consistency group408, while maintaining the snapshot sets430,432within the consistency group408. Unlike conventional data storage systems, the LUN414can be removed from the consistency group408without having to delete the LUN snapshot414(1) and the LUN snapshot416(1) in the snapshot set430, and/or the LUN snapshot416(2) and the LUN snapshot420(2) in the snapshot set432. This is because the data storage system provides (i) the expanded snapshot metadata414.1for the LUN snapshot414(1), (ii) the expanded snapshot metadata416.1for the LUN snapshot416(1), (iii) the expanded snapshot metadata416.2for the LUN snapshot416(2), (iv) the expanded snapshot metadata420.2for the LUN snapshot420(2), and (v) the enhanced consistency group data structure408bfor the consistency group408, which together allow relationships among the LUN snapshots414(1),416(1),416(2),420(2), the snapshot sets430,432, the consistency group408, and the LUNs414,416,420to be maintained in the storage device array400while the LUN414is being removed from the consistency group408, as well as after the LUN414has been removed from the consistency group408. It is noted that, once the LUN414has been removed from the consistency group408, only read requests pertaining to the LUN snapshot414(1) will be acted upon by the data storage system. In one embodiment, a delete operation pertaining to the LUN snapshot414(1) will be performed by the data storage system in the event that (i) the snapshot set430is deleted, refreshed, or restored, (ii) the consistency group408is removed from the storage device array400, or (iii) the LUN414is removed from the storage device array400.

FIG. 4cdepicts the storage device array400of the storage appliance within the data storage system (such as the data storage system104) at a third PIT T3(also referred to herein as the “time T3”). At the time T3, the data storage system operates to remove the LUN414from the storage device array400(as illustrated by an “X” on the LUN414), to remove the LUN416from the consistency group408, and to add the LUN416to the consistency group410. In addition, the data storage system operates to create a snapshot set434containing a LUN snapshot420(3) of the LUN420taken at the time T3, and to create a snapshot set436containing a LUN snapshot416(3) of the LUN416and a LUN snapshot418(3) of the LUN418, each taken at the time T3.

FIG. 4c(continued) depicts the expanded snapshot metadata414.1for the LUN snapshot414(1), the expanded snapshot metadata416.1for the LUN snapshot416(1), the expanded snapshot metadata420.1for the LUN snapshot420(1), the expanded snapshot metadata414.2for the LUN snapshot414(2), the expanded snapshot metadata416.2for the LUN snapshot416(2), and the expanded snapshot metadata420.2for the LUN snapshot420(2), as well as expanded snapshot metadata420.3for the LUN snapshot420(3) in the snapshot set434, and expanded snapshot metadata416.3and expanded snapshot metadata418.3for the LUN snapshot416(3) and the LUN snapshot418(3), respectively, in the snapshot set436. The expanded snapshot metadata416.3for the LUN snapshot416(3) includes an SSID436.1for the snapshot set436, the CGID410.1for the consistency group410, and other snapshot metadata418.7. The expanded snapshot metadata418.3for the LUN snapshot418(3) includes the SSID436.1for the snapshot set436, the CGID410.1for the consistency group410, and other snapshot metadata418.8. The expanded snapshot metadata420.3for the LUN snapshot420(3) includes an SSID434.1for the snapshot set434, the CGID408.1for the consistency group408, and other snapshot metadata418.9.

FIG. 4c(continued) further depicts an enhanced consistency group data structure408cfor the consistency group408that includes a CG data structure419.3and a CGSSL440.3. The CGSSL440.3includes three (3) nodes, namely, the CGSSN450.1, the CGSSN450.2, and a CGSSN450.3. The CGSSN450.1contains the SSID430.1for the snapshot set430and the pointer424.1to the storage volume list406.1. The CGSSN450.2contains the SSID432.1for the snapshot set432and the pointer424.2to the storage volume list406.2. The CGSSN450.3contains the SSID434.1for the snapshot set434, and a pointer424.3to a storage volume list406.3of the name of the LUN420that has the associated LUN snapshot420(3) in the snapshot set434. As shown inFIG. 4c(continued), the storage volume list406.3can include the worldwide name (WWN) of the LUN420. In addition,FIG. 4c(continued) depicts an enhanced consistency group data structure410afor the consistency group410that includes a CG data structure419.4and a CGSSL440.4. The CGSSL440.4includes a single node, namely, a CGSSN450.4, which contains the SSID436.1for the snapshot set436, and a pointer424.4to a storage volume list406.4of the names of the LUN416and the LUN418, in which each of the LUNs416,418has its associated LUN snapshot416(3),418(3), respectively, in the snapshot set436. As shown inFIG. 4c(continued), the storage volume list406.4can include the worldwide name (WWN) of the LUN416, and a worldwide name (WWN) of the LUN418.

As described herein with reference toFIG. 4c, at the time T3, the data storage system operates to remove the LUN414from the storage device array400(as illustrated by an “X” on the LUN414). Before removing the LUN414, the data storage system accesses the expanded snapshot metadata414.1for the LUN snapshot414(1) to determine the values of the CGID408.1and the SSID430.1. Having determined the values of the CGID408.1and the SSID430.1, the data storage system accesses the CGSSN450.1of the CGSSL440.3within the enhanced consistency group data structure408c, navigates to the storage volume list406.1using the pointer424.1, and deletes the name of the LUN414from the storage volume list406.1(as illustrated by an “X” on the WWN LUN414). Further, the data storage system removes the LUN snapshot414(1) from the snapshot set430of the consistency group408(as illustrated by an “X” on the LUN snapshot414(1)), and deletes the expanded snapshot metadata414.1for the LUN snapshot414(1) (as illustrated by an “X” on the expanded snapshot metadata414.1). The data storage system also accesses the expanded snapshot metadata414.2for the LUN snapshot414(2) to determine the values of the CGID and the SSID. Having determined that the values of the CGID and the SSID are each the default value of 0xFFFF, the data storage system determines that the LUN snapshot414(2) is a standalone LUN snapshot, removes the LUN snapshot414(2) from the storage device array400(as illustrated by an “X” on the LUN snapshot414(2)), and deletes the expanded snapshot metadata414.2for the LUN snapshot414(2) (as illustrated by an “X” on the expanded snapshot metadata414.2).

In this example, the data storage system operates to remove the LUN416from the consistency group408, and to add the LUN416to the consistency group410, while maintaining the snapshot sets430,432,434within the consistency group408. Unlike conventional data storage systems, the LUN416can be removed from the consistency group408(or added to the consistency group410) without having to delete the LUN snapshot414(1) and the LUN snapshot416(1) in the snapshot set430, to delete the LUN snapshot416(2) and the LUN snapshot420(2) in the snapshot set432, and/or to delete the LUN snapshot420(3) in the snapshot set434. This is because the data storage system provides (i) the expanded snapshot metadata414.1for the LUN snapshot414(1), (ii) the expanded snapshot metadata416.1for the LUN snapshot416(1), (iii) the expanded snapshot metadata416.2for the LUN snapshot416(2), (iv) the expanded snapshot metadata420.2for the LUN snapshot420(2), (v) the expanded snapshot metadata420.3for the LUN snapshot420(3), and (vi) the enhanced consistency group data structure408cfor the consistency group408, which together allow relationships among the LUN snapshots414(1),416(1),416(2),420(2),420(3), the snapshot sets430,432,434, the consistency group408, and the LUNs414,416,420to be maintained in the storage device array400while the LUN416is being removed from the consistency group408(or being added to the consistency group410), as well as after the LUN416has been removed from the consistency group408(or has been added to the consistency group410). It is noted that, once the LUN416has been added to the consistency group410, only read requests pertaining to the LUN snapshot416(1) and the LUN snapshot416(2) will be acted upon by the data storage system. In one embodiment, a delete operation pertaining to the LUN snapshot416(1) will be performed by the data storage system in the event that (i) the snapshot set430is deleted, refreshed, or restored, (ii) the consistency group408is removed from the storage device array400, or (iii) the LUN416is removed from the storage device array400. Similarly, a delete operation pertaining to the LUN snapshot416(2) will be performed by the data storage system in the event that (i) the snapshot set432is deleted, refreshed, or restored, (ii) the consistency group408is removed from the storage device array400, or (iii) the LUN416is removed from the storage device array400.

An exemplary method of implementing snapshot sets for consistency groups of storage volumes in a data storage system is described below with reference toFIG. 5. As depicted in block502, a consistency group of storage volumes is formed on a data storage system, in which the consistency group has a data structure. As depicted in block504, a set of snapshots of the storage volumes is created in the consistency group. As depicted in block506, snapshot metadata is generated for each snapshot in the set of snapshots, in which the snapshot metadata includes a first identifier of the set of snapshots and a second identifier of the consistency group. As depicted in block508, a snapshot set list including at least one snapshot set node is incorporated into the data structure of the consistency group, in which the snapshot set node includes the first identifier of the set of snapshots, and a pointer to a storage volume list of the storage volumes having the respective snapshots in the set of snapshots. As depicted in block510, one or more of (i) removing a respective storage volume from among the storage volumes in the consistency group, and (ii) adding an additional storage volume to the consistency group, is performed without deleting any of the snapshots in the set of snapshots.

Having described the above illustrative embodiments, other alternative embodiments or variations may be made and/or practiced. For example, it was described herein that snapshots of storage volumes in a snapshot set of a consistency group could be deleted, refreshed, and/or restored. In certain embodiments, if a storage volume that has a snapshot in the snapshot set is removed from the consistency group, and a refresh operation is performed on the snapshot set, then (i) the snapshot of the removed storage volume is deleted from the snapshot set (and the name of the removed storage volume is deleted from the list of storage volumes having snapshots in the snapshot set), and (ii) the refresh operation if performed on the remaining snapshots in the snapshot set. In further embodiments, if a delete operation is performed on the snapshot set, then all of the snapshots in the snapshot set are deleted (and the CGSSN that includes the SSID of the snapshot set, as well as the storage volume list pointed to by the pointer included in the CGSSN, are deleted from the CGSSL for the consistency group).

In other embodiments, if a restore operation is to be performed on the snapshot set that includes the snapshot of the removed storage volume, then a user of the data storage system is provided, such as via a graphical user interface (GUI) of the data storage system, the option of (i) discarding the snapshot of the removed storage volume, or (ii) adding the removed storage volume back to the consistency group. If the user chooses to discard the snapshot of the removed storage volume, then (i) the snapshot of the removed storage volume is deleted from the snapshot set (and the name of the removed storage volume is deleted from the list of storage volumes having snapshots in the snapshot set), and (ii) the restore operation is performed on the remaining snapshots in the snapshot set. If the user chooses to add the removed storage volume back to the consistency group, then the restore operation is performed on all of the snapshots in the snapshot set, including the snapshot of the storage volume that was previously removed from and subsequently added back to the consistency group (and the CGSSN that includes the SSID of the snapshot set, as well as the storage volume list pointed to by the pointer included in the CGSSN, are updated to reflect the addition of the storage volume to the consistency group).

It is noted that the disclosed systems and methods or portions thereof may be embodied as a computer program product including one or more non-transient, computer-readable storage media, such as a magnetic disk, magnetic tape, compact disk, DVD, optical disk, flash drive, solid state drive (SSD), Secure Digital (SD) chip or device, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), and/or the like. Any number of computer-readable media may be used. The media may be encoded with instructions that, when executed on one or more computers or other processors, perform the process or processes described herein. Such media may be considered articles of manufacture or machines, and may be transportable from one machine to another.