Source: http://www.google.com/patents/US7228456?dq=%E2%80%9Cconfiguration+using+structure+and+rules+to+provide+a+user+interface.%E2%80%9D&ei=ANUpTrT8BsTm0QHVpJX-Cg
Timestamp: 2014-11-23 23:55:35
Document Index: 625726589

Matched Legal Cases: ['art 140', 'art 200', 'art 440', 'art 440', 'art 500', 'art 1000', 'art 1000', 'art 1030']

Patent US7228456 - Data recovery for virtual ordered writes for multiple storage devices - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inAdvanced Patent SearchPatentsRecovering data provided in chunks to a plurality of secondary storage devices includes, for each of the secondary storage devices, discarding data corresponding chunks for which all data thereof has not been received, and, for each of the secondary storage devices, restoring a chunk of data thereto...http://www.google.com/patents/US7228456?utm_source=gb-gplus-sharePatent US7228456 - Data recovery for virtual ordered writes for multiple storage devicesAdvanced Patent SearchPublication numberUS7228456 B2Publication typeGrantApplication numberUS 10/724,670Publication dateJun 5, 2007Filing dateDec 1, 2003Priority dateDec 1, 2003Fee statusPaidAlso published asUS20050132248Publication number10724670, 724670, US 7228456 B2, US 7228456B2, US-B2-7228456, US7228456 B2, US7228456B2InventorsDouglas E. LeCrone, Kevin C. Heasley, Vadim Longinov, Mark J. Halstead, David MeiriOriginal AssigneeEmc CorporationExport CitationBiBTeX, EndNote, RefManPatent Citations (23), Non-Patent Citations (6), Referenced by (4), Classifications (14), Legal Events (2) External Links: USPTO, USPTO Assignment, EspacenetData recovery for virtual ordered writes for multiple storage devicesUS 7228456 B2Abstract Recovering data provided in chunks to a plurality of secondary storage devices includes, for each of the secondary storage devices, discarding data corresponding chunks for which all data thereof has not been received, and, for each of the secondary storage devices, restoring a chunk of data thereto where all of the chunks of data restored to the plurality of secondary storage devices correspond to a particular transmission cycle of primary storage devices that provide data to the plurality of secondary storage devices. Recovering data may also include, following discarding and prior to restoring, for each of the plurality of secondary storage devices having two different chunks, waiting for external intervention to indicate whether to restore a particular one of the chunks. The external intervention may be provided by a host computer that is proximate to at least one of the secondary storage devices or may be provided by a host computer that is proximate to at least one of the primary storage computers.
1. A method of recovering data provided by a plurality of primary storage devices to a plurality of secondary storage devices, comprising:
the secondary storage devices receiving data in chunks, each chunk having a sequence number associated therewith, wherein writes by the primary storage devices begun before a particular time are assigned a first sequence number and writes begun by the primary storage devices begun after the particular time are assigned a second sequence number different than the first sequence number and wherein switching of sequence numbers is coordinated between the primary storage devices;
for each of the secondary storage devices, discarding data corresponding to chunks for which all data thereof has not been received; and
for each of the secondary storage devices, restoring a chunk of data thereto wherein all of the chunks of data restored to the plurality of secondary storage devices have the same sequence number.
restoring most recent chunks for all of the plurality of secondary storage devices in response to there being two different chunks associated with all of the plurality of secondary storage devices, wherein a first one of the two chunks corresponds to a first sequence number and wherein a second one of the two chunks corresponding to a different sequence number.
for each of the secondary storage devices, restoring a chunk of data corresponding to a particular sequence number wherein all of the secondary storage devices contain a chunk of data corresponding to the particular sequence number.
9. A method, according to claim 1, wherein each sequence number is assigned a particular tag value that is provided with each chunk of data.
10. A method, according to claim 9, wherein the tag values are used to determine the particular sequence number for each of the chunks of data.
11. Computer-readable medium containing computer software that recovers data provided by a plurality of primary storage devices to a plurality of secondary storage devices, comprising:
executable code that receives data in chunks, each chunk having a sequence number associated therewith, wherein writes by the primary storage devices begun before a particular time are assigned a first sequence number and writes begun by the primary storage devices begun after the particular time are assigned a second sequence number different than the first sequence number and wherein switching of sequence numbers is coordinated between the primary storage devices;
executable code that discards data corresponding to chunks for which all data thereof has not been received for each of the secondary storage devices; and
executable code that restores a chunk of data thereto for each of the secondary storage devices, wherein all of the chunks of data restored to the plurality of secondary storage devices have the same sequence number.
12. Computer-readable medium, according to claim 11, further comprising:
13. Computer-readable medium software, according to claim 12, further comprising:
executable code that restores most recent chunks for all of the plurality of secondary storage devices in response to there being two different chunks associated with all of the plurality of secondary storage devices, wherein a first one of the two chunks corresponds to a first sequence number and wherein a second one of the two chunks corresponding to a different sequence number.
14. Computer-readable medium, according to claim 13, further comprising:
15. Computer-readable medium, according to claim 12, further comprising:
executable code that restores a chunk of data corresponding to a particular sequence number wherein all of the secondary storage devices contain a chunk of data corresponding to the particular sequence number.
16. Computer-readable medium, according to claim 15, further comprising:
17. Computer-readable medium, according to claim 11, wherein each sequence number is assigned a particular tag value that is provided with each chunk of data.
18. Computer-readable medium, according to claim 17, wherein the tag values are used to determine the particular sequence number for each of the chunks of data.
In some instances, it may be desirable to copy data from one storage device to another. For example, if a host writes data to a first storage device, it may be desirable to copy that data to a second storage device provided in a different location so that if a disaster occurs that renders the first storage device inoperable, the host (or another host) may resume operation using the data of the second storage device. Such a capability is provided, for example, by the Remote Data Facility (RDF) product provided by EMC Corporation of Hopkinton, Mass. With RDF, a first storage device, denoted the �primary storage device� (or �R1�) is coupled to the host. One or more other storage devices, called �secondary storage devices� (or �R2�) receive copies of the data that is written to the primary storage device by the host. The host interacts directly with the primary storage device, but any data changes made to the primary storage device are automatically provided to the one or more secondary storage devices using RDF. The primary and secondary storage devices may be connected by a data link, such as an ESCON link, a Fibre Channel link, and/or a Gigabit Ethernet link. The RDF functionality may be facilitated with an RDF adapter (RA) provided at each of the storage devices.
SUMMARY OF THE INVENTION According to the present invention, recovering data provided in chunks to a plurality of secondary storage devices includes, for each of the secondary storage devices, discarding data corresponding chunks for which all data thereof has not been received, and, for each of the secondary storage devices, restoring a chunk of data thereto where all of the chunks of data restored to the plurality of secondary storage devices correspond to a particular transmission cycle of primary storage devices that provide data to the plurality of secondary storage devices. Recovering data may also include, following discarding and prior to restoring, for each of the plurality of secondary storage devices having two different chunks, waiting for external intervention to indicate whether to restore a particular one of the chunks. The external intervention may be provided by a host computer that is proximate to at least one of the secondary storage devices or may be provided by a host computer that is proximate to at least one of the primary storage computers. Recovering data may also include restoring most recent chunks for all of the plurality of secondary storage devices in response to there being two different chunks associated with all of the plurality of secondary storage devices, where a first one of the two chunks corresponds to a first transmission cycle and where a second one of the two chunks corresponding to a different transmission cycle. Recovering data may also include discarding chunks that are not restored. Recovering data may also include, for each of the secondary storage devices, restoring a chunk of data corresponding to a particular transmission cycle where all of the secondary storage devices contain a chunk of data corresponding to the particular transmission cycle. Recovering data may also include discarding chunks that are not restored. Each transmission cycle may be assigned a particular tag value that is provided with each chunk of data. The tag values may be used to determine the particular cycle for each of the chunks of data.
DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS Referring to FIG. 1, a diagram 20 shows a relationship between a host 22, a local storage device 24 and a remote storage device 26. The host 22 reads and writes data from and to the local storage device 24 via a host adapter (HA) 28, which facilitates the interface between the host 22 and the local storage device 24. Although the diagram 20 only shows one host 22 and one HA 28, it will be appreciated by one of ordinary skill in the art that multiple HA's may be used and that one or more HA's may have one or more hosts coupled thereto.
The local storage device 24 includes a first plurality of RDF adapter units (RA's) 30 a, 30 b, 30 c and the remote storage device 26 includes a second plurality of RA's 32 a�32 c. The RA's 30 a�30 c, 32 a�32 c are coupled to the RDF link 29 and are similar to the host adapter 28, but are used to transfer data between the storage devices 24, 26. The software used in connection with the RA's 30 a�30 c, 32 a�32 c is discussed in more detail hereinafter.
The storage devices 24, 26 may include one or more disks, each containing a different portion of data stored on each of the storage devices 24, 26. FIG. 1 shows the storage device 24 including a plurality of disks 33 a, 33 b, 33 c and the storage device 26 including a plurality of disks 34 a, 34 b, 34 c. The RDF functionality described herein may be applied so that the data for at least a portion of the disks 33 a�33 c of the local storage device 24 is copied, using RDF, to at least a portion of the disks 34 a�34 c of the remote storage device 26. It is possible that other data of the storage devices 24, 26 is not copied between the storage devices 24, 26, and thus is not identical.
Each of the disks 33 a�33 c is coupled to a corresponding disk adapter unit (DA) 35 a, 35 b, 35 c that provides data to a corresponding one of the disks 33 a�33 c and receives data from a corresponding one of the disks 33 a�33 c. Similarly, a plurality of DA's 36 a, 36 b, 36 c of the remote storage device 26 are used to provide data to corresponding ones of the disks 34 a�34 c and receive data from corresponding ones of the disks 34 a�34 c. An internal data path exists between the DA's 35 a�35 c, the HA 28 and the RA's 30 a�30 c of the local storage device 24. Similarly, an internal data path exists between the DA's 36 a�36 c and the RA's 32 a�32 c of the remote storage device 26. Note that, in other embodiments, it is possible for more than one disk to be serviced by a DA and that it is possible for more than one DA to service a disk.
The local storage device 24 also includes a global memory 37 that may be used to facilitate data transferred between the DA's 35 a�35 c, the HA 28 and the RA's 30 a�30 c. The memory 37 may contain tasks that are to be performed by one or more of the DA's 35 a�35 c, the HA 28 and the RA's 30 a�30 c, and a cache for data fetched from one or more of the disks 33 a�33 c. Similarly, the remote storage device 26 includes a global memory 38 that may contain tasks that are to be performed by one or more of the DA's 36 a�36 c and the RA's 32 a�32 c, and a cache for data fetched from one or more of the disks 34 a�34 c. Use of the memories 37, 38 is described in more detail hereinafter.
The storage space in the local storage device 24 that corresponds to the disks 33 a�33 c may be subdivided into a plurality of volumes or logical devices. The logical devices may or may not correspond to the physical storage space of the disks 33 a�33 c. Thus, for example, the disk 33 a may contain a plurality of logical devices or, alternatively, a single logical device could span both of the disks 33 a, 33 b. Similarly, the storage space for the remote storage device 26 that comprises the disks 34 a�34 c may be subdivided into a plurality of volumes or logical devices, where each of the logical devices may or may not correspond to one or more of the disks 34 a�34 c. Providing an RDF mapping between portions of the local storage device 24 and the remote storage device 26 involves setting up a logical device on the remote storage device 26 that is a remote mirror for a logical device on the local storage device 24. The host 22 reads and writes data from and to the logical device on the local storage device 24 and the RDF mapping causes modified data to be transferred from the local storage device 24 to the remote storage device 26 using the RA's, 30 a�30 c, 32 a�32 c and the RDF link 29. In steady state operation, the logical device on the remote storage device 26 contains data that is identical to the data of the logical device on the local storage device 24. The logical device on the local storage device 24 that is accessed by the host 22 is referred to as the �R1 volume� (or just �R1�) while the logical device on the remote storage device 26 that contains a copy of the data on the R1 volume is called the �R2 volume� (or just �R2�). Thus, the host reads and writes data from and to the R1 volume and RDF handles automatic copying and updating of the data from the R1 volume to the R2 volume.
Referring to FIG. 3, a diagram 70 illustrates items used to construct and maintain the chunks 52, 54. A standard logical device 72 contains data written by the host 22 and corresponds to the data element 51 of FIG. 2 and the disks 33 a�33 c of FIG. 1. The standard logical device 72 contains data written by the host 22 to the local storage device 24.
Two linked lists of pointers 74, 76 are used in connection with the standard logical device 72. The linked lists 74, 76 correspond to data that may be stored, for example, in the memory 37 of the local storage device 24. The linked list 74 contains a plurality of pointers 81�85, each of which points to a slot of a cache 88 used in connection with the local storage device 24. Similarly, the linked list 76 contains a plurality of pointers 91�95, each of which points to a slot of the cache 88. In some embodiments, the cache 88 may be provided in the memory 37 of the local storage device 24. The cache 88 contains a plurality of cache slots 102�104 that may be used in connection to writes to the standard logical device 72 and, at the same time, used in connection with the linked lists 74, 76.
Each of the linked lists 74, 76 may be used for one of the chunks of data 52, 54 so that, for example, the linked list 74 may correspond to the chunk of data 52 for sequence number N while the linked list 76 may correspond to the chunk of data 54 for sequence number N−1. Thus, when data is written by the host 22 to the local storage device 24, the data is provided to the cache 88 and, in some cases (described elsewhere herein), an appropriate pointer of the linked list 74 is created. Note that the data will not be removed from the cache 88 until the data is destaged to the standard logical device 72 and the data is also no longer pointed to by one of the pointers 81�85 of the linked list 74, as described elsewhere herein.
In an embodiment herein, one of the linked lists 74, 76 is deemed �active� while the other is deemed �inactive�. Thus, for example, when the sequence number N is even, the linked list 74 may be active while the linked list 76 is inactive. The active one of the linked lists 74, 76 handles writes from the host 22 while the inactive one of the linked lists 74, 76 corresponds to the data that is being transmitted from the local storage device 24 to the remote storage device 26.
While the data that is written by the host 22 is accumulated using the active one of the linked lists 74, 76 (for the sequence number N), the data corresponding to the inactive one of the linked lists 74, 76 (for previous sequence number N−1) is transmitted from the local storage device 24 to the remote storage device 26. The RA's 30 a�30 c use the linked lists 74, 76 to determine the data to transmit from the local storage device 24 to the remote storage device 26.
Referring to FIG. 4, a slot 120, like one of the slots 102�104 of the cache 88, includes a header 122 and data 124. The header 122 corresponds to overhead information used by the system to manage the slot 120. The data 124 is the corresponding data from the disk that is being (temporarily) stored in the slot 120. Information in the header 122 includes pointers back to the disk, time stamp(s), etc.
The header 122 also includes a cache stamp 126 used in connection with the system described herein. In an embodiment herein, the cache stamp 126 is eight bytes. Two of the bytes are a �password� that indicates whether the slot 120 is being used by the system described herein. In other embodiments, the password may be one byte while the following byte is used for a pad. As described elsewhere herein, the two bytes of the password (or one byte, as the case may be) being equal to a particular value indicates that the slot 120 is pointed to by at least one entry of the linked lists 74, 76. The password not being equal to the particular value indicates that the slot 120 is not pointed to by an entry of the linked lists 74, 76. Use of the password is described elsewhere herein.
Processing begins at a first step 142 where a slot corresponding to the write is locked. In an embodiment herein, each of the slots 102�104 of the cache 88 corresponds to a track of data on the standard logical device 72. Locking the slot at the step 142 prevents additional processes from operating on the relevant slot during the processing performed by the HA 28 corresponding to the steps of the flow chart 140.
Referring to FIG. 6, a flow chart 200 illustrates steps performed in connection with the RA's 30 a�30 c scanning the inactive one of the lists 72, 74 to transmit RDF data from the local storage device 24 to the remote storage device 26. As discussed above, the inactive one of the lists 72, 74 points to slots corresponding to the N−1 cycle for the R1 device when the N cycle is being written to the R1 device by the host using the active one of the lists 72, 74.
If it is determined at the step 202 that there is data available for sending, control transfers from the step 202 to a step 204, where the slot is verified as being correct. The processing performed at the step 204 is an optional �sanity check� that may include verifying that the password field is correct and verifying that the sequence number field is correct. If there is incorrect (unexpected) data in the slot, error processing may be performed, which may include notifying a user of the error and possibly error recovery processing.
Referring to FIG. 7, a diagram 240 illustrates creation and manipulation of the chunks 56, 58 used by the remote storage device 26. Data that is received by the remote storage device 26, via the link 29, is provided to a cache 242 of the remote storage device 26. The cache 242 may be provided, for example, in the memory 38 of the remote storage device 26. The cache 242 includes a plurality of cache slots 244�246, each of which may be mapped to a track of a standard logical storage device 252. The cache 242 is similar to the cache 88 of FIG. 3 and may contain data that can be destaged to the standard logical storage device 252 of the remote storage device 26. The standard logical storage device 252 corresponds to the data element 62 shown in FIG. 2 and the disks 34 a�34 c shown in FIG. 1.
The remote storage device 26 also contains a pair of cache only virtual devices 254, 256. The cache only virtual devices 254, 256 corresponded device tables that may be stored, for example, in the memory 38 of the remote storage device 26. Each track entry of the tables of each of the cache only virtual devices 254, 256 point to either a track of the standard logical device 252 or point to a slot of the cache 242. Cache only virtual devices are described in a copending U.S. patent application titled VIRTUAL STORAGE DEVICE THAT USES VOLATILE MEMORY, filed on Mar. 25, 2003, now U.S. Pat. No. 7,113,945, which is incorporated by reference herein.
The plurality of cache slots 244�246 may be used in connection to writes to the standard logical device 252 and, at the same time, used in connection with the cache only virtual devices 254, 256. In an embodiment herein, each of track table entry of the cache only virtual devices 254, 256 contain a null to indicate that the data for that track is stored on a corresponding track of the standard logical device 252. Otherwise, an entry in the track table for each of the cache only virtual devices 254, 256 contains a pointer to one of the slots 244�246 in the cache 242.
Each of the cache only virtual devices 254, 256 corresponds to one of the data chunks 56, 58. Thus, for example, the cache only virtual device 254 may correspond to the data chunk 56 while the cache only virtual device 256 may correspond to the data chunk 58. In an embodiment herein, one of the cache only virtual devices 254, 256 may be deemed �active� while the other one of the cache only virtual devices 254, 256 may be deemed �inactive�. The inactive one of the cache only virtual devices 254, 256 may correspond to data being received from the local storage device 24 (i.e., the chunk 56) while the active one of the cache only virtual device 254, 256 corresponds to data being restored (written) to the standard logical device 252.
Data from the local storage device 24 that is received via the link 29 may be placed in one of the slots 244�246 of the cache 242. A corresponding pointer of the inactive one of the cache only virtual devices 254, 256 may be set to point to the received data. Subsequent data having the same sequence number may be processed in a similar manner. At some point, the local storage device 24 provides a message committing all of the data sent using the same sequence number. Once the data for a particular sequence number has been committed, the inactive one of the cache only virtual devices 254, 256 becomes active and vice versa. At that point, data from the now active one of the cache only virtual devices 254, 256 is copied to the standard logical device 252 while the inactive one of the cache only virtual devices 254, 256 is used to receive new data (having a new sequence number) transmitted from the local storage device 24 to the remote storage device 26.
Following the step 324 is a step 326 where the slot pointer for the standard logical device 252 is changed to point to the slot in the cache. Following the step 326 is a test step 328 which determines if the operations performed at the steps 322, 324, 326 have been successful. In some instances, a single operation called a �compare and swap� operation may be used to perform the steps 322, 324, 326. If these operations are not successful for any reason, then control transfers from the step 328 back to the step 308 to reexamine if the corresponding track of the standard logical device 252 is in the cache. Otherwise, if it is determined at the test step 328 that the previous operations have been successful, then control transfers from the test step 328 to the step 318, discussed above.
Following the step 358 is a step 362 where the copying of data for the inactive one of the lists 74, 76 is suspended. As discussed elsewhere herein, the inactive one of the lists is scanned to send corresponding data from the local storage device 24 to the remote storage device 26. It is useful to suspend copying data until the sequence number switch is completed. In an embodiment herein, the suspension is provided by sending a message to the RA's 30 a�30 c. However, it will be appreciated by one of ordinary skill in the art that for embodiments that use other components to facilitate sending data using the system described herein, suspending copying may be provided by sending appropriate messages/commands to the other components.
Referring to FIG. 11, a diagram 400 illustrates items used to construct and maintain the chunks 52, 54. A standard logical device 402 contains data written by the host 22 and corresponds to the data element 51 of FIG. 2 and the disks 33 a�33 c of FIG. 1. The standard logical device 402 contains data written by the host 22 to the local storage device 24.
The cache 408 contains a plurality of cache slots 412�414 that may be used in connection to writes to the standard logical device 402 and, at the same time, used in connection with the cache only virtual devices 404, 406. In an embodiment herein, each track table entry of the cache only virtual devices 404, 406 contains a null to point to a corresponding track of the standard logical device 402. Otherwise, an entry in the track table for each of the cache only virtual devices 404, 406 contains a pointer to one of the slots 412�414 in the cache 408.
In an embodiment herein, one of the cache only virtual devices 404, 406 is deemed �active� while the other is deemed �inactive�. Thus, for example, when the sequence number N is even, the cache only virtual device 404 may be active while the cache only virtual device 406 is inactive. The active one of the cache only virtual devices 404, 406 handles writes from the host 22 while the inactive one of the cache only virtual devices 404, 406 corresponds to the data that is being transmitted from the local storage device 24 to the remote storage device 26.
While the data that is written by the host 22 is accumulated using the active one of the cache only virtual devices 404, 406 (for the sequence number N), the data corresponding to the inactive one of the cache only virtual devices 404, 406 (for previous sequence number N−1) is transmitted from the local storage device 24 to the remote storage device 26. For this and related embodiments, the DA's 35 a�35 c of the local storage device handle scanning the inactive one of the cache only virtual devices 404, 406 to send copy requests to one or more of the RA's 30 a�30 c to transmit the data from the local storage device 24 to the remote storage device 26. Thus, the steps 362, 374, discussed above in connection with suspending and resuming copying, may include providing messages/commands to the DA's 35 a�35 c. Once the data has been transmitted to the remote storage device 26, the corresponding entry in the inactive one of the cache only virtual devices 404, 406 may be set to null. In addition, the data may also be removed from the cache 408 (i.e., the slot returned to the pool of slots for later use) if the data in the slot is not otherwise needed for another purpose (e.g., to be destaged to the standard logical device 402). A mechanism may be used to ensure that data is not removed from the cache 408 until all mirrors (including the cache only virtual devices 404, 406) are no longer using the data. Such a mechanism is described, for example, in U.S. Pat. No. 5,537,568 issued on Jul. 16, 1996 and in U.S. patent application Ser. No. 09/850,551 filed on Jul. 7, 2001, both of which are incorporated by reference herein.
Referring to FIG. 12, a flow chart 440 illustrates steps performed by the HA 28 in connection with a host 22 performing a write operation for embodiments where two COVD's are used by the R1 device to provide the system described herein. Processing begins at a first step 442 where a slot corresponding to the write is locked. In an embodiment herein, each of the slots 412�414 of the cache 408 corresponds to a track of data on the standard logical device 402. Locking the slot at the step 442 prevents additional processes from operating on the relevant slot during the processing performed by the HA 28 corresponding to the steps of the flow chart 440.
Referring to FIG. 13, a flow chart 500 illustrates steps performed in connection with the local storage device 24 transmitting the chunk of data 54 to the remote storage device 26. The transmission essentially involves scanning the inactive one of the cache only virtual devices 404, 406 for tracks that have been written thereto during a previous iteration when the inactive one of the cache only virtual devices 404, 406 was active. In this embodiment, the DA's 35 a�35 c of the local storage device 24 scan the inactive one of the cache only virtual devices 404, 406 to copy the data for transmission to the remote storage device 26 by one or more of the RA's 30 a�30 c using the RDF protocol.
Following the step 506 or the step 505 if the slot is a direct slot is a step 507 where data being sent (directly or indirectly from the slot) is copied by one of the DA's 35 a�35 c to be sent from the local storage device 24 to the remote storage device 26 using the RDF protocol. Following the step 507 is a test step 508 where it is determined if the remote storage device 26 has acknowledged receipt of the data. If not, then control transfers from the step 508 back to the step 507 to resend the data. In other embodiments, different and more involved processing may used to send data and acknowledge receipt thereof. Such processing may include error reporting and alternative processing that is performed after a certain number of attempts to send the data have failed.
Referring to FIG. 14, a diagram 700 illustrates a host 702 coupled to a plurality of local storage devices 703�705. The diagram 700 also shows a plurality of remote storage devices 706�708. Although only three local storage devices 703�705 and three remote storage devices 706�708 are shown in the diagram 700, the system described herein may be expanded to use any number of local and remote storage devices.
Each of the local storage devices 703�705 is coupled to a corresponding one of the remote storage devices 706�708 so that, for example, the local storage device 703 is coupled to the remote storage device 706, the local storage device 704 is coupled to the remote storage device 707 and the local storage device 705 is coupled to the remote storage device 708. The local storage device is 703�705 and remote storage device is 706�708 may be coupled using the ordered writes mechanism described herein so that, for example, the local storage device 703 may be coupled to the remote storage device 706 using the ordered writes mechanism. As discussed elsewhere herein, the ordered writes mechanism allows data recovery using the remote storage device in instances where the local storage device and/or host stops working and/or loses data.
In some instances, the host 702 may run a single application that simultaneously uses more than one of the local storage devices 703�705. In such a case, the application may be configured to insure that application data is consistent (recoverable) at the local storage devices 703�705 if the host 702 were to cease working at any time and/or if one of the local storage devices 703�705 were to fail. However, since each of the ordered write connections between the local storage devices 703�705 and the remote storage devices 706�708 is asynchronous from the other connections, then there is no assurance that data for the application will be consistent (and thus recoverable) at the remote storage devices 706�708. That is, for example, even though the data connection between the local storage device 703 and the remote storage device 706 (a first local/remote pair) is consistent and the data connection between the local storage device 704 and the remote storage device 707 (a second local/remote pair) is consistent, it is not necessarily the case that the data on the remote storage devices 706, 707 is always consistent if there is no synchronization between the first and second local/remote pairs.
For applications on the host 702 that simultaneously use a plurality of local storage devices 703�705, it is desirable to have the data be consistent and recoverable at the remote storage devices 706�708. This may be provided by a mechanism whereby the host 702 controls cycle switching at each of the local storage devices 703�705 so that the data from the application running on the host 702 is consistent and recoverable at the remote storage devices 706�708. This functionality is provided by a special application that runs on the host 702 that switches a plurality of the local storage devices 703�705 into multi-box mode, as described in more detail below.
Referring to FIG. 15, a table 730 has a plurality of entries 732�734. Each of the entries 732�734 correspond to a single local/remote pair of storage devices so that, for example, the entry 732 may correspond to pair of the local storage device 703 and the remote storage device 706, the entry 733 may correspond to pair of the local storage device 704 and the remote storage device 707 and the entry 734 may correspond to the pair of local storage device 705 and the remote storage device 708. Each of the entries 732�734 has a plurality of fields where a first field 736 a�736 c represents a serial number of the corresponding local storage device, a second field 738 a�738 c represents a session number used by the multi-box group, a third field 742 a�742 c represents the serial number of the corresponding remote storage device of the local/remote pair, and a fourth field 744 a�744 c represents the session number for the multi-box group. The table 730 is constructed and maintained by the host 702 in connection with operating in multi-box mode. In addition, the table 730 is propagated to each of the local storage devices and the remote storage devices that are part of the multi-box group. The table 730 may be used to facilitate recovery, as discussed in more detail below.
Once it is determined at the test step 786 that all of the local/remote pairs in the multi-box group are ready to switch, control transfers from the step 786 to a step 788 where an index variable, N, is set equal to one. The index variable N is used to iterate through all the local/remote pairs (i.e., all of the entries 732�734 of the table 730 of FIG. 15). Following the step 788 is a test step 792 which determines if the index variable, N, is greater than the number of local/remote pairs in the multi-box group. If not, then control transfers from the step 792 to a step 794 where an open window is performed for the Nth local storage device of the Nth pair by the host sending a command (e.g., an appropriate system command) to the Nth local storage device. Opening the window for the Nth local storage device at the step 794 causes the Nth local storage device to suspend writes so that any write by a host that is not begun prior to opening the window at the step 794 will not be completed until the window is closed (described below). Not completing a write operation prevents a second dependant write from occurring prior to completion of the cycle switch. Any writes in progress that were begun before opening the window may complete prior to the window being closed.
Referring to FIG. 22, a diagram 980 illustrates the host 702, local storage devices 703�705 and remote storage devices 706�708, that are shown in the diagram 700 of FIG. 14. The Diagram 980 also includes a first alternative host 982 that is coupled to the host 702 and the local storage devices 703�705. The diagram 980 also includes a second alternative host 984 that is coupled to the remote storage devices 706�708. The alternative hosts 982, 984 may be used for data recovery, as described in more detail below.
When recovery of data at the remote site is necessary, the recovery may be performed by the host 702 or, by the host 982 provided that the links between the local storage devices 703�705 and the remote storage devices 706�708 are still operational. If the links are not operational, then data recovery may be performed by the second alternative host 984 that is coupled to the remote storage devices 706�708. The second alternative host 984 may be provided in the same location as one or more of the remote storage devices 706�708. Alternatively, the second alternative host 984 may be remote from all of the remote storage devices 706�708. The table 730 that is propagated throughout the system is accessed in connection with data recovery to determine the members of the multi-box group.
Referring to FIG. 23, a flow chart 1000 illustrates steps performed by each of the remote storage devices 706�708 in connection with the data recovery operation. The steps of the flowchart 1000 may be executed by each of the remote storage devices 706�708 upon receipt of a signal or a message indicating that data recovery is necessary. In some embodiments, it may be possible for a remote storage device to automatically sense that data recovery is necessary using, for example, conventional criteria such as length of time since last write.
Referring to FIG. 24, a flow chart 1030 illustrates steps performed in connection with one of the hosts 702, 982, 984 determining whether to discard or restore each of the inactive chunks of each of the remote storage devices. The one of the hosts 702, 982, 984 that is performing the restoration communicates with the remote storage devices 706�708 to provide commands thereto and to receive information therefrom using the tags that are assigned by the host as discussed elsewhere herein.
Following execution of the step 1044, each of the remote storage devices contains data associated with the same tag value as data for the other ones of the remote storage devices. Accordingly, the recovered data on the remote storage devices 706�708 should be consistent.
Patent CitationsCited PatentFiling datePublication dateApplicantTitleUS5086502 *Nov 13, 1989Feb 4, 1992Intelligence Quotient International LimitedMethod of operating a data processing systemUS5206939Sep 24, 1990Apr 27, 1993Emc CorporationSystem and method for disk mapping and data retrievalUS5369767May 17, 1989Nov 29, 1994International Business Machines Corp.Servicing interrupt requests in a data processing system without using the services of an operating systemUS5469453 *Feb 21, 1995Nov 21, 1995Mti Technology CorporationData corrections applicable to redundant arrays of independent disksUS5537568Dec 12, 1994Jul 16, 1996Emc CorporationSystem for dynamically controlling cache manager maintaining cache index and controlling sequential data accessUS5623599 *Jul 29, 1994Apr 22, 1997International Business Machines CorporationMethod and apparatus for processing a synchronizing marker for an asynchronous remote data copyUS5742792May 28, 1996Apr 21, 1998Emc CorporationRemote data mirroringUS5778394Dec 23, 1996Jul 7, 1998Emc CorporationSpace reclamation system and method for use in connection with tape logging systemUS5845147Mar 19, 1996Dec 1, 1998Emc CorporationSingle lock command for an I/O storage system that performs both locking and I/O data operationUS5857208May 31, 1996Jan 5, 1999Emc CorporationMethod and apparatus for performing point in time backup operation in a computer systemUS5901327 *Mar 17, 1997May 4, 1999Emc CorporationBundling of write data from channel commands in a command chain for transmission over a data link between data storage systems for remote data mirroringUS6088694 *Mar 31, 1998Jul 11, 2000International Business Machines CorporationContinuous availability and efficient backup for externally referenced objectsUS6157991Apr 1, 1998Dec 5, 2000Emc CorporationMethod and apparatus for asynchronously updating a mirror of a source deviceUS6209002Feb 17, 1999Mar 27, 2001Emc CorporationMethod and apparatus for cascading data through redundant data storage unitsUS6301643Sep 3, 1998Oct 9, 2001International Business Machines CorporationMulti-environment data consistencyUS6415292 *Sep 28, 2001Jul 2, 2002Emc CorporationShuttle-based mechanism for numbering concurrent chains of independent data transfersUS6502205Nov 10, 2000Dec 31, 2002Emc CorporationAsynchronous remote data mirroring systemUS6732124 *Feb 9, 2000May 4, 2004Fujitsu LimitedData processing system with mechanism for restoring file systems based on transaction logsUS6769074 *May 21, 2001Jul 27, 2004Lumigent Technologies, Inc.System and method for transaction-selective rollback reconstruction of database objectsUS6898685Mar 25, 2003May 24, 2005Emc CorporationOrdering data writes from a local storage device to a remote storage deviceUS6970987 *Jan 27, 2003Nov 29, 2005Hewlett-Packard Development Company, L.P.Method for storing data in a geographically-diverse data-storing system providing cross-site redundancyUS7054883Dec 1, 2003May 30, 2006Emc CorporationVirtual ordered writes for multiple storage devicesUS20020023199Oct 12, 2001Feb 21, 2002Voom Technologies, Inc.Computer storage restoration* Cited by examinerNon-Patent CitationsReference1Article re: Network Appliance(TM) SnapMirror(R) Software ; found on Network Appliance, Inc. web page; www.netapp.com/ftp/snapmirror.pdf.2U.S. Appl. No. 09/850,551, filed May 7, 2001, by Josef Ezra.3U.S. Appl. No. 10/224,918, filed Aug. 21, 2002, by Douglas E. LeCrone, et al.4U.S. Appl. No. 10/225,021, filed Aug. 21, 2002, by Douglas E. LeCrone, et al.5U.S. Appl. No. 10/396,786, filed Mar. 25, 2003, by David Meiri, et al.6U.S. Appl. No. 10/396,800, filed Mar. 25, 2003, by Hana Moreshet, et al.Referenced byCiting PatentFiling datePublication dateApplicantTitleUS7376764Dec 10, 2002May 20, 2008Emc CorporationMethod and apparatus for migrating data in a computer systemUS7661028 *Dec 18, 2006Feb 9, 2010Commvault Systems, Inc.Rolling cache configuration for a data replication systemUS7870355Dec 18, 2006Jan 11, 2011Commvault Systems, Inc.Log based data replication system with disk swapping below a predetermined rateUS8190948 *Jul 31, 2008May 29, 2012Emc CorporationHandling out of order I/O using temporal relationships of transmitted data* Cited by examinerClassifications U.S. Classification714/15, 714/E11.106, 714/E11.102, 714/16, 714/18, 714/6.23International ClassificationG06F11/00, G06F11/08Cooperative ClassificationG06F11/2071, G06F11/2082, G06F11/2064European ClassificationG06F11/20S2S, G06F11/20S2P, G06F11/20S2ELegal EventsDateCodeEventDescriptionDec 6, 2010FPAYFee paymentYear of fee payment: 4Dec 1, 2003ASAssignmentOwner name: EMC CORPORATION, MASSACHUSETTSFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LECRONE, DOUGLAS E.;HEASLEY, KEVIN C.;LONGINOV, VADIM;AND OTHERS;REEL/FRAME:014766/0831Effective date: 20031201RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services©2012 Google