Source: http://www.google.com/patents/US8078813?dq=5754119
Timestamp: 2017-05-24 07:16:06
Document Index: 342622449

Matched Legal Cases: ['art 1230', 'art 1230', 'art 1300', 'art 1230', 'art 1330', 'art 1390', 'art 1450', 'art 1470', 'art 1320']

Patent US8078813 - Triangular asynchronous replication - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inPatentsStoring recovery data includes providing chunks of data to a remote destination, where each chunk of data represents data written before a first time and after a second time and where the second time for one of the particular chunks corresponds to a first time for a subsequent one of the particular chunks,...http://www.google.com/patents/US8078813?utm_source=gb-gplus-sharePatent US8078813 - Triangular asynchronous replicationAdvanced Patent SearchTry the new Google Patents, with machine-classified Google Scholar results, and Japanese and South Korean patents.Publication numberUS8078813 B2Publication typeGrantApplication numberUS 10/955,470Publication dateDec 13, 2011Filing dateSep 30, 2004Priority dateSep 30, 2004Fee statusPaidAlso published asCN1779660A, CN100428190C, US8185708, US9558083, US20060069887, US20060069893, US20120036333Publication number10955470, 955470, US 8078813 B2, US 8078813B2, US-B2-8078813, US8078813 B2, US8078813B2InventorsDouglas E. LeCrone, Gary H. Cox, Brett A. Quinn, David Meiri, Mark J. Halstead, Benjamin W. YoderOriginal AssigneeEmc CorporationExport CitationBiBTeX, EndNote, RefManPatent Citations (35), Non-Patent Citations (2), Referenced by (6), Classifications (14), Legal Events (5) External Links: USPTO, USPTO Assignment, EspacenetTriangular asynchronous replication
The storage space in the local storage device 24 that corresponds to the disks 33 a-33 c may be subdivided into a plurality of volumes or logical devices. The logical devices may or may not correspond to the physical storage space of the disks 33 a-33 c. Thus, for example, the disk 33 a may contain a plurality of logical devices or, alternatively, a single logical device could span both of the disks 33 a, 33 b. Similarly, the storage space for the remote storage device 26 that comprises the disks 34 a-34 c may be subdivided into a plurality of volumes or logical devices, where each of the logical devices may or may not correspond to one or more of the disks 34 a-34 c. Providing an RDF mapping between portions of the local storage device 24 and the remote storage device 26 involves setting up a logical device on the remote storage device 26 that is a remote mirror for a logical device on the local storage device 24. The host 22 reads and writes data from and to the logical device on the local storage device 24 and the RDF mapping causes modified data to be transferred from the local storage device 24 to the remote storage device 26 using the RA's, 30 a-30 c, 32 a-32 c and the RDF link 29. In steady state operation, the logical device on the remote storage device 26 contains data that is identical to the data of the logical device on the local storage device 24. The logical device on the local storage device 24 that is accessed by the host 22 is referred to as the “R1 volume” (or just “R1”) while the logical device on the remote storage device 26 that contains a copy of the data on the R1 volume is called the “R2 volume” (or just “R2”). Thus, the host reads and writes data from and to the R1 volume and RDF handles automatic copying and updating of the data from the R1 volume to the R2 volume. The system described herein may be implemented using software, hardware, and/or a combination of software and hardware where software may be stored in an appropriate storage medium (i.e., a computer-readable storage medium) and executed by one or more processors.
Processing for the flow chart 1230 begins at a first step 1232 where a first SDDF session, SDDF—1, is created. In an embodiment described herein, creation of an SDDF session does not cause automatic activation of the session. Following step 1232 is a step 1234 where the bits of the bitmap of the SDDF session created at the step 1232 are cleared. Following step 1234 is a step 1236 where a second SDDF session, SDDF—2, is created. Following step 1236 is a step 1238 where the bits of the bitmap of the SDDF session created at the step 1236 are cleared.
Following the step 1238 is a step 1242 where a state is initialized. The state initialized at the step 1242 may be used to determine which of the SDDF sessions, SDDF—1 or SDDF—2, will be activated. As described in more detail elsewhere herein, there may be two possible states and the state set at the step 1242 may be toggled to cause the SDDF—1 session and the SDDF—2 session to be alternatively activated. In other embodiments, a token or some other type of variable may be used to indicate the selection of either SDDF—1 or SDDF—2. Following the step 1242 is a step 1244 where SDDF—1 is activated. Activating SDDF—1 at the step 1244 causes the bits of the bit map of the SDDF—1 session to be set whenever a track (or other data increment) of the local destination 1204 is modified.
The SDDF—1 and SDDF—2 sessions are used by the local destination 1204 to keep track of the active and inactive buffers used by the source group 1202 in connection with ordered writes by the source group 1202 to the remote destination 1206. As discussed in more detail elsewhere herein, each time the source group 1202 makes a cycle switch in connection with ordered writes from the source group 1202 to the remote destination 1206, the source group 1202 sends a message to the local destination 1204 indicating that a cycle switch has been performed so that the local destination 1204 may toggle the state (initialized in the step 1242, discussed above). Use of the cycle switch information by the local destination 1204 is discussed in more detail elsewhere herein.
If the received data indicates that the local group 1202 is ready to switch, then control transfers from the step 1254 to a step 1256, where it is determined if the inactive one of the SDDF sessions (SDDF—1 or SDDF—2) is clear. In some embodiments, the SDDF sessions may be cleared at the step 1256. In other instances, the amount of time needed to clear an SDDF session at the step 1256 would be unacceptable, in which case more than two SDDF sessions may be used for SDDF—1 and SDDF—2 and may be rotated so that an SDDF session that is about to be activated is always cleared asynchronously. In any event, the processing performed at the step 1256 relates to clearing the inactive one of SDDF—1 and SDDF—2 so that, after performing the step 1256, the inactive session is clear.
Following the step 1256 is a step 1258 where the inactive one of the SDDF sessions is activated so that both SDDF—1 and SDDF—2 are activated after performing the processing at the step 1258. Thus, subsequent writes reflected in the bitmaps for both SDDF—1 and SDDF—2. Following the step 1258, processing is complete.
If it is determined at the step 1254 that the received data does not correspond to a ready to switch signal, then control transfers from the step 1254 to a test step 1262 where it is determined if the received data corresponds to a cycle switch being performed. If so, then control transfers from the step 1262 to a step 1264 where the state, initialized at the step 1242 of the flow chart 1230 of FIG. 27, is toggled. As discussed elsewhere herein, the state is used to determine which one of SDDF—1 and SDDF—2 will be activated and deactivated. Following the step 1264 is a step 1266, where one of the SDDF sessions, SDDF—1 or SDDF—2, is deactivated, depending on the particular value of the state set at the step 1264. Note that even though an SDDF session is deactivated at the step 1266, that SDDF session is not cleared until the next ready to switch signal is received. Of course, if more than two SDDF sessions are used for SDDF—1 and SDDF—2, as discussed above, then the SDDF session deactivated at the step 1266 may be maintained while another SDDF session is cleared to prepare for being activated at the step 1258, discussed above.
If it is determined at the step 1262 that the received data does not correspond to a cycle switch, then control transfers from the test step 1262 to a step 1268 where the I/O is performed. For example, if the I/O is a write operation, then, at the step 1268, data is written to the storage area of the local destination 1204. Following step 1268 is a step 1272 where it is determined if the I/O operation is a write operation. If not (e.g., the I/O operation is a read operation), then processing is complete. Otherwise, control transfers from the step 1272 to a step 1274 where a bit is set in the appropriate one of the SDDF sessions, SDDF—1, SDDF—2, or both depending upon which one of the SDDF sessions is activated. Following step 1274, processing is complete.
In some instances, it may not be desirable to wait to clear an SDDF bitmap just prior to pointing the same SDDF bitmap. In those cases, it may be useful to have more than two SDDF bitmaps where two at a time are used like SDDF—1 and SDDF—2 while the remainder of the SDDF bitmaps are already clear and waiting to be used or are being cleared using a background process. For example, using three bitmaps SDDF_A, SDDF_B, and SDDF_C, SDDF—1 may correspond to SDDF_A while SDDF—2 may correspond to SDDF_C. In such a case, SDDF_B may be cleared while processing is being performed on SDDF_A and SDDF_C. When the cycle switches, SDDF_B (which is already clear) may be used while SDDF_C is cleared using a background process that may run even after the cycle switch is complete and new data is being logged to SDDF_B.
Referring to FIG. 28C, a flow chart 1300 illustrates yet another embodiment for processing related to the local destination 1204 receiving an I/O from the source group 1202 during normal (i.e., non-failure) operation. Processing begins at a first step 1302 where the I/O is received by the local destination 1204. Following the step 1302 is a test step 1304 where it is determined if the received data corresponds to a cycle switch being performed. If so, then control transfers from the step 1304 to a test step 1306 where it is determined if two or more cycle switches have occurred since the last time the state was toggled. If not, then processing is complete. Otherwise, control transfers from the step 1306 to a step 1307 where it is determined if the currently inactive SDDF session, SDDF_X, is clear. If so, then control transfers from the step 1307 to a step 1308 where the state, initialized at the step 1242 of the flow chart 1230 of FIG. 27, is toggled. As discussed elsewhere herein, the state is used to determine which one of SDDF—1 and SDDF—2 will be activated and deactivated.
Following the step 1308 is a step 1309 where one of the SDDF sessions, SDDF—1 or SDDF—2, as indicated by the state, is activated. Following the step 1309 is a step 1312 where the other one of the SDDF sessions is deactivated. Following the step 1312, processing is complete.
If it is determined at the step 1304 that the received data does not correspond to a cycle switch, then control transfers from the test step 1304 to a step 1316 where the I/O is performed. For example, if the I/O is a write operation, then, at the step 1316 data is written to the storage area of the local destination 1204. Following step 1316 is a step 1317 where it is determined if the I/O operation is a write operation. If not (e.g., the I/O operation is a read operation), then processing is complete. Otherwise, control transfers from the step 1317 to a step 1318 where a bit is set in the appropriate one of the SDDF sessions, SDDF—1 or SDDF—2, (SDDF_X or SDDF_Y) depending upon which one of the SDDF sessions is activated. Following step 1318, processing is complete.
Processing begins at a first step 1322 where a third SDDF session, SDDF—3 is created. Following the step 1322 is a step 1324 where the bitmap of the SDDF session created at the step 1322 is cleared. Following step 1324 is a step 1326 where a token value (described in more detail elsewhere herein) is set to zero. Following the step 1326, processing is complete.
Referring to FIG. 30, a flow chart 1330 illustrates steps performed by the remote destination 1206 in connection with collection of recovery data. Processing begins at a first step 1331 where the remote destination 1206 waits for a failure message from the source group 1202 or from some other source, as appropriate. Once a failure message has been received, control transfers from the step 1331 to a step 1332 where SDDF—3 session is activated to begin collecting data regarding the tracks (or other appropriate data increments) of the remote destination 1206 to which a write has been performed. Note, however, that SDDF—3 reflects writes that have been committed (i.e., are one behind the current cycle being received).
If it is determined at the test step 1336 that the data from the source group 1202 indicates a cycle switch, then control transfers from the test step 1336 to a step 1338 to increment the token, which keeps track of the number of cycle switch since beginning collection of recovery data. Following the step 1338 is a step 1342 where the bitmap of the SDDF—3 is set to reflect data of the committed cycle that had been written. Note that the processing at the step 1342 may be performed automatically in connection with operation of the activated SDDF sessions, in which case in may not be necessary to make the step 1342 part of the recovery process shown if FIG. 30.
Following the step 1366 is a test step 1368 where it is determined if the token value (discussed above) is greater than a predetermined value N. Note that the token value indicates the number of cycle switches that have occurred since collection of error recovery data began at the remote destination 1206. If the link between the source group 1202 and the local destination 1204 has not been working and the remote destination 1206 has begun collection of recovery data, then the remote destination 1206 may contain more up-to-date data than the local destination 1204. This will be determined by looking at the value of the token, which indicates the number of cycle switches that have occurred since the remote destination 1206 received a signal to begin collecting recovery data. Thus, if it is determined at the test step 1368 that the token is greater than some pre-determined value N (e.g., two), then control transfers from the test step 1368 to a step 1371, where the bitmaps for all of the SDDF sessions (SDDF—1, SDDF—2, and SDDF—3) are ORed (using an inclusive OR) to determine the tracks (or other data amounts) of the remote destination 1206 and possibly of the local destination 1204 that correspond to data for the active and inactive buffers sent or in transit between the source group 1202 and the remote destination 1206 prior to failure of the source group as well as possible data that may be different on the local destination 1204.
Following the step 1371 is a step 1372 where the remote destination 1206 sends data from the tracks corresponding to the set bits of the bitmap that was the result or ORing the three bitmaps for SDDF—1, SDDF—2, and SDDF—3. The data from these tracks may be copied to the local destination 1204 so that the remote destination 1206 and the local destination 1204 may be synchronized. Following the step 1372, processing is complete. In an embodiment herein, N may be set to be no lower than two. Also, note that it may be possible to resume operation with a host coupled to an appropriate one of the local destination 1204 or the remote destination 1206 prior to completion of the copies initiated at the step 1376 or at the step 1372.
If it is determined at the test step 1368 that the token does not have a value greater than N (e.g., the token is zero), then control transfers from the test step 1368 to a step 1374 where the bitmaps for all of the SDDF sessions (SDDF—1, SDDF—2, and, if it exists, SDDF—3) are ORed (using an inclusive OR) to determine the tracks (or other data amounts) of the local destination 1204 that correspond to data for the active and inactive buffers sent or in transit between the source group 1202 and the remote destination 1206 prior to failure of the source group 1202. Following the step 1374 is a step 1376 where the data corresponding to the ORing of the bitmaps is sent from the local destination 1204 to the remote destination 1206 via the communication link 1208. Once the data is sent from the local destination 1204 to the remote destination 1206, then the local destination 1204 and the remote destination 1206 will be synchronized. Following the step 1376, processing is complete.
Referring to FIG. 33, a flow chart 1390 illustrates in more detail steps performed in connection with the step 1376 where data is copied from the local destination 1204 to the remote destination 1206 or the step 1372 where data is copied from the remote destination 1206 to the local destination 1204. Processing begins at a first step 1392 where the OR of SDDF—1, SDDF—2, and SDDF—3 (from the step 1374) is used to set a device table at whichever one of the local destination 1204 and the remote destination 1206 will be the R1 device after recovery. If data is to be copied from the R1 device to the R2 device, then the device table locations corresponding to remote tracks are set at the step 1392. Otherwise, if data is to be copied from the R2 device to the R1 device, then the device table locations corresponding to local tracks are set at the step 1392. In many instances, the tracks corresponding to the modification of the table at the step 1392 will be the same or a superset of the modification to the table from the step 1386, discussed above.
Referring to FIG. 34, a flow chart 1450 illustrates steps performed in connection with reinitializing the recovery parameters once normal operation is restored between the source group 1202, the local destination 1204, and the remote destination 1206. Processing begins at a first step 1452 where both of the SDDF sessions, SDDF—1 and SDDF—2, are deactivated. Following the step 1452 is a step 1454 where SDDF—1 is cleared. Following step 1454 is a step 1456 where SDDF—2 is cleared. Following the step 1456 is a step 1458 with a pointer that points to one of the SDDF sessions is made to point SDDF—1. Following step 1458 is a step 1462 where SDDF—1 is activated. Following step 1462, processing is complete.
Referring to FIG. 35, a flow chart 1470 illustrates steps performed in connection with resetting recovery parameters used by the remote destination 1206. Processing begins at a first step 1472 where SDDF—3 is deactivated. Following the step 1472 is a step 1474 where SDDF—3 is cleared. Following the step 1474 is a step 1476 where the token used by the remote destination 1206 is cleared (set to zero). Following the step 1476, processing is complete. Note that, in some embodiments, it is possible to also terminate SDDF—3 at or after the step 1472 so that SDDF—3 may be recreated at the step 1322 of the flow chart 1320 of FIG. 29, discussed above.
Patent CitationsCited PatentFiling datePublication dateApplicantTitleUS5206939Sep 24, 1990Apr 27, 1993Emc CorporationSystem and method for disk mapping and data retrievalUS5537568Dec 12, 1994Jul 16, 1996Emc CorporationSystem for dynamically controlling cache manager maintaining cache index and controlling sequential data accessUS5742792May 28, 1996Apr 21, 1998Emc CorporationRemote data mirroringUS5778394Dec 23, 1996Jul 7, 1998Emc CorporationSpace reclamation system and method for use in connection with tape logging systemUS5778430Apr 19, 1996Jul 7, 1998Eccs, Inc.Method and apparatus for computer disk cache managementUS5845147Mar 19, 1996Dec 1, 1998Emc CorporationSingle lock command for an I/O storage system that performs both locking and I/O data operationUS5857208May 31, 1996Jan 5, 1999Emc CorporationMethod and apparatus for performing point in time backup operation in a computer systemUS6014676May 16, 1998Jan 11, 2000Fairbanks Systems GroupSystem and method for backing up computer files over a wide area computer networkUS6366986Jun 30, 1998Apr 2, 2002Emc CorporationMethod and apparatus for differential backup in a computer storage systemUS6446175Jul 28, 1999Sep 3, 2002Storage Technology CorporationStoring and retrieving data on tape backup system located at remote storage system siteUS6594742May 7, 2001Jul 15, 2003Emc CorporationCache management via statistically adjusted slot agingUS6862632Nov 30, 2001Mar 1, 2005Emc CorporationDynamic RDF system for transferring initial data between source and destination volume wherein data maybe restored to either volume at same time other data is writtenUS7082506 *Mar 8, 2002Jul 25, 2006Hitachi, Ltd.Remote copy control method, storage sub-system with the method, and large area data storage system using themUS7426609Feb 13, 2003Sep 16, 2008Fujitsu LimitedDisk control apparatus, disk control method, remote disk control apparatus, and remote disk control methodUS20010000818Dec 22, 2000May 3, 2001Teruo NagasawaSubsystem replacement methodUS20010007102Dec 19, 2000Jul 5, 2001Mathieu GagneMethod and apparatus for cascading data through redundant data storage unitsUS20020112135Jan 29, 2002Aug 15, 2002AlcatelMethod of storing computer data and a corresponding storage deviceUS20030051111Mar 8, 2002Mar 13, 2003Hitachi, Ltd.Remote copy control method, storage sub-system with the method, and large area data storage system using themUS20030131278Jan 10, 2002Jul 10, 2003Hitachi, Ltd.Apparatus and method for multiple generation remote backup and fast restoreUS20040039888Aug 21, 2002Feb 26, 2004Lecrone Douglas E.Storage automated replication processingUS20040039959Aug 21, 2002Feb 26, 2004Lecrone Douglas E.SAR restart and going home proceduresUS20040073831Dec 27, 2002Apr 15, 2004Moshe YanaiRemote data mirroringUS20050038968 *Oct 2, 2003Feb 17, 2005Takashige IwamuraMulti-site remote-copy systemUS20050203972 *Mar 12, 2004Sep 15, 2005Cochran Robert A.Data synchronization for two data mirrorsUS20050251633 *May 4, 2004Nov 10, 2005Micka William FApparatus, system, and method for synchronizing an asynchronous mirror volume using a synchronous mirror volumeDE19900331A1Jan 7, 1999Jul 8, 1999Nat Semiconductor CorpUniversal data bus endpoint moduleEP0158409A2Jan 21, 1985Oct 16, 1985The British Library BoardImproved threshold detectorEP1158409A2Sep 15, 2000Nov 28, 2001Hitachi, Ltd.Storage system making possible data synchronisation confirmation at time of asynchronous remote copyEP1229448A2Jan 9, 2002Aug 7, 2002Hewlett-Packard CompanyRedundant backup device and methodEP1507205A2Oct 2, 2003Feb 16, 2005Hitachi, Ltd.Multi-site remote-copy systemEPI158409A2 Title not availableGB2306022A Title not availableJP2003122509A Title not availableJP2003131917A Title not availableWO2002017055A1Aug 18, 2000Feb 28, 2002Fujitsu LimitedDisk control device, disk control method, remote disk control device, and remote disk control method* Cited by examinerNon-Patent CitationsReference1N. Takahashi, et al., "An Improved Inter-Disk Remote Copying Method to Cope with Disasters," Transactions of IEICE D, The Institute of Electronics, Information and Communication Engineers (IEICE), Japan, Jun. 1, 2004, vol. J87-D-I, No. 6, pp. 675-692.2U.S. Appl. No. 10/396,800, filed Mar. 25, 2003, Moreshet et al.Referenced byCiting PatentFiling datePublication dateApplicantTitleUS8959305Jun 29, 2012Feb 17, 2015Emc CorporationSpace reclamation with virtually provisioned devicesUS9100330 *Jul 13, 2012Aug 4, 2015Emc CorporationIntroduction of read delay or write delay in servers of a geographically distributed data processing system so that clients read up-to-date dataUS9130826 *Mar 15, 2013Sep 8, 2015Enterasys Networks, Inc.System and related method for network monitoring and control based on applicationsUS9275117Dec 6, 2012Mar 1, 2016Emc CorporationFast dependency mining using access patterns in a storage systemUS9584393Mar 15, 2013Feb 28, 2017Extreme Networks, Inc.Device and related method for dynamic traffic mirroring policyUS20140280889 *Mar 15, 2013Sep 18, 2014Enterasys Networks, Inc.System and related method for network monitoring and control based on applications* Cited by examinerClassifications U.S. Classification711/162, 709/217International ClassificationG06F13/00Cooperative ClassificationG06F11/2082, G06F11/2079, G06F11/2074, G06F11/2071, G06F11/2069, G06F11/2058European ClassificationG06F11/20S2P6, G06F11/20S2P2, G06F11/20S2M, G06F11/20S2C, G06F11/20S2SLegal EventsDateCodeEventDescriptionSep 30, 2004ASAssignmentOwner name: EMC CORPORATION, MASSACHUSETTSFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LECRONE, DOUGLAS E.;COX, GARY H.;YODER, BENJAMIN W.;REEL/FRAME:015864/0335Effective date: 20040929Dec 2, 2004ASAssignmentOwner name: EMC CORPORATION, MASSACHUSETTSFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:QUINN, BRETT A.;MEIRI, DAVID;HALSTEAD, MARK J.;REEL/FRAME:016032/0100Effective date: 20041101Jun 15, 2015FPAYFee paymentYear of fee payment: 4Sep 21, 2016ASAssignmentOwner name: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATFree format text: SECURITY AGREEMENT;ASSIGNORS:ASAP SOFTWARE EXPRESS, INC.;AVENTAIL LLC;CREDANT TECHNOLOGIES, INC.;AND OTHERS;REEL/FRAME:040134/0001Effective date: 20160907Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AFree format text: SECURITY AGREEMENT;ASSIGNORS:ASAP SOFTWARE EXPRESS, INC.;AVENTAIL LLC;CREDANT TECHNOLOGIES, INC.;AND OTHERS;REEL/FRAME:040136/0001Effective date: 20160907Sep 29, 2016ASAssignmentOwner name: EMC IP HOLDING COMPANY LLC, MASSACHUSETTSFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:EMC CORPORATION;REEL/FRAME:040203/0001Effective date: 20160906RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services