Switching between virtual ordered writes mode and synchronous or semi-synchronous RDF transfer mode

Transitioning from a first data transfer mode to a second data transfer mode that is different from the first data transfer mode includes a primary storage device transitioning to the second data transfer mode by first transferring writes associated with a first chunk of data to a secondary storage device after completion of all writes associated with the first chunk of data and then, after all writes associated with the first chunk of data have been transferred to the secondary storage device, transferring writes associated with a second chunk of data to the secondary storage device using the first data transfer mode. Writes begun after initiating transitioning are provided to the secondary storage device using the second data transfer mode.

BACKGROUND OF THE INVENTION

1. Technical Field

This application relates to computer storage devices, and more particularly to the field of transferring data between storage devices.

2. Description of Related Art

Host processor systems may store and retrieve data using a storage device containing a plurality of host interface units (host adapters), disk drives, and disk interface units (disk adapters). Such storage devices are provided, for example, by EMC Corporation of Hopkinton, Mass. and disclosed in U.S. Pat. No. 5,206,939 to Yanai et al., U.S. Pat. No. 5,778,394 to Galtzur et al., U.S. Pat. No. 5,845,147 to Vishlitzky et al., and U.S. Pat. No. 5,857,208 to Ofek. The host systems access the storage device through a plurality of channels provided therewith. Host systems provide data and access control information through the channels to the storage device and the storage device provides data to the host systems also through the channels. The host systems do not address the disk drives of the storage device directly, but rather, access what appears to the host systems as a plurality of logical disk units. The logical disk units may or may not correspond to the actual disk drives. Allowing multiple host systems to access the single storage device unit allows the host systems to share data stored therein.

In some instances, it may be desirable to copy data from one storage device to another. For example, if a host writes data to a first storage device, it may be desirable to copy that data to a second storage device provided in a different location so that if a disaster occurs that renders the first storage device inoperable, the host (or another host) may resume operation using the data of the second storage device. Such a capability is provided, for example, by the Remote Data Facility (RDF) product provided by EMC Corporation of Hopkinton, Mass. With RDF, a first storage device, denoted the “primary storage device” (or “R1”) is coupled to the host. One or more other storage devices, called “secondary storage devices” (or “R2”) receive copies of the data that is written to the primary storage device by the host. The host interacts directly with the primary storage device, but any data changes made to the primary storage device are automatically provided to the one or more secondary storage devices using RDF. The primary and secondary storage devices may be connected by a data link, such as an ESCON link, a Fibre Channel link, and/or a Gigabit Ethernet link. The RDF functionality may be facilitated with an RDF adapter (RA) provided at each of the storage devices.

RDF allows synchronous data transfer where, after data written from a host to a primary storage device is transferred from the primary storage device to a secondary storage device using RDF, receipt is acknowledged by the secondary storage device to the primary storage device which then provides a write acknowledge back to the host. Thus, in synchronous mode, the host does not receive a write acknowledge from the primary storage device until the RDF transfer to the secondary storage device has been completed and acknowledged by the secondary storage device.

A drawback to the synchronous RDF system is that the latency of each of the write operations is increased by waiting for the acknowledgement of the RDF transfer. This problem is worse when there is a long distance between the primary storage device and the secondary storage device; because of transmission delays, the time delay required for making the RDF transfer and then waiting for an acknowledgement back after the transfer is complete may be unacceptable.

It is also possible to use RDF in an a semi-synchronous mode, in which case the data is written from the host to the primary storage device which acknowledges the write immediately and then, at the same time, begins the process of transferring the data to the secondary storage device. Thus, for a single transfer of data, this scheme overcomes some of the disadvantages of using RDF in the synchronous mode. However, for data integrity purposes, the semi-synchronous transfer mode does not allow the primary storage device to transfer data to the secondary storage device until a previous transfer is acknowledged by the secondary storage device. Thus, the bottlenecks associated with using RDF in the synchronous mode are simply delayed by one iteration because transfer of a second amount of data cannot occur until transfer of previous data has been acknowledged by the secondary storage device.

Another possibility is to have the host write data to the primary storage device in asynchronous mode and have the primary storage device copy data to the secondary storage device in the background. The background copy involves cycling through each of the tracks of the primary storage device sequentially and, when it is determined that a particular block has been modified since the last time that block was copied, the block is transferred from the primary storage device to the secondary storage device. Although this mechanism may attenuate the latency problem associated with synchronous and semi-synchronous data transfer modes, a difficulty still exists because there can not be a guarantee of data consistency between the primary and secondary storage devices. If there are problems, such as a failure of the primary system, the secondary system may end up with out-of-order changes that make the data unusable.

A proposed solution to this problem is the Symmetrix Automated Replication (SAR) process, which is described in pending U.S. patent application Ser. Nos. 10/224,918 and 10/225,021, both of which were filed on Aug. 21, 2002. The SAR uses devices (BCV's) that can mirror standard logical devices. A BCV device can also be split from its standard logical device after being mirrored and can be resynced (i.e., reestablished as a mirror) to the standard logical devices after being split. However, using the SAR process requires the significant overhead of continuously splitting and resyncing the BCV's. The SAR process also uses host control and management, which relies on the controlling host being operational. In addition, the cycle time for a practical implementation of a SAR process is on the order of twenty to thirty minutes, and thus the amount of data that may be lost when an RDF link and/or primary device fails could be twenty to thirty minutes worth of data.

Thus, it would be desirable to have an RDF system that exhibits some of the beneficial qualities of each of the different techniques discussed above while reducing the drawbacks. Such a system would exhibit low latency for each host write regardless of the distance between the primary device and the secondary device and would provide consistency (recoverability) of the secondary device in case of failure.

SUMMARY OF THE INVENTION

According to the present invention, transitioning from a first data transfer mode to a second data transfer mode that is different from the first data transfer mode, includes a primary storage device receiving a plurality of data writes while being in the first data transfer mode, the primary storage device associating data writes begun after a first time and before a second time with a first chunk of data, the primary storage device associating data writes begun after the second time with a second chunk of data different from the first chunk of data, and the primary storage device transitioning to the second data transfer mode after the second time by first transferring writes associated with the first chunk of data to a secondary storage device after completion of all writes associated with the first chunk of data and then, after all writes associated with the first chunk of data have been transferred to the secondary storage device, transferring writes associated with the second chunk of data to the secondary storage device using the first data transfer mode, where writes begun after initiating transitioning are provided to the secondary storage device using the second data transfer mode. Transitioning from a first data transfer mode to a second data transfer mode that is different from the first data transfer mode may also include, following the primary storage device transferring all writes associated with the first chunk of data to the secondary storage device, the primary storage device sending a message to the secondary storage device, and, in response to receiving the message from the primary storage device, the secondary storage device storing the data writes associated with the first chunk of data. Transitioning from a first data transfer mode to a second data transfer mode that is different from the first data transfer mode may also include, after storing all of the data writes associated with the first chunk of data, the secondary storage device sending an acknowledge to the primary storage device. In the second data transfer mode, the primary storage device may acknowledge a write thereto in response to the secondary storage device acknowledging receipt of data corresponding to the write. In the second data transfer mode, the primary storage device may acknowledge a write thereto in response to the secondary storage device acknowledging receipt of data previously written to a storage location thereof corresponding to the write. The primary storage device may acknowledge the write prior to the secondary storage device acknowledging receipt of data corresponding to the write. Transitioning from a first data transfer mode to a second data transfer mode that is different from the first data transfer mode may also include, following transitioning to the second data transfer mode, the primary storage device transferring writes associated with the second chunk of data to the secondary storage device. A subset of the writes associated with the second chunk may be transferred using the second data transfer mode. Transitioning from a first data transfer mode to a second data transfer mode that is different from the first data transfer mode may also include, prior to transitioning from the first data transfer mode to the second data transfer mode, inhibiting creation of additional chunks of data. Transitioning from a first data transfer mode to a second data transfer mode that is different from the first data transfer mode may also include the primary storage device sending a message to the secondary storage device indicating the transitioning from the first data transfer mode to the second data transfer mode. Transitioning from a first data transfer mode to a second data transfer mode that is different from the first data transfer mode may also include, in response to a data write occurring prior to transferring all writes associated with the first chunk of data to the secondary storage device, associating the write with the second chunk of data. Transitioning from a first data transfer mode to a second data transfer mode that is different from the first data transfer mode may also include, in response to a data write occurring after transferring all writes associated with the first chunk of data to the secondary storage device and before transferring all writes associated with the second chunk of data to the secondary storage device, merging the write with data in the second chunk of data if the write corresponds to data already in the second chunk of data. Transitioning from a first data transfer mode to a second data transfer mode that is different from the first data transfer mode may also include, in response to a data write occurring after transferring all writes associated with the first chunk of data to the secondary storage device and before transferring all writes associated with the second chunk of data to the secondary storage device, transferring the data using the second data transfer mode if the write does not correspond to data already in the second chunk of data. Transitioning from a first data transfer mode to a second data transfer mode that is different from the first data transfer mode may also include providing a transition variable that is periodically incremented to indicate a state of the transition, wherein the transition variable is used to select the first mode or the second mode for writes that occur after initiation of the transition.

According further to the present invention, storing data provided to a remote storage device includes receiving a first plurality of writes associated with a first chunk of data, receiving a second plurality of writes associated with a second chunk of data, wherein the second plurality of writes are all begun after the first plurality of writes, receiving a message indicating a transition from a first data transfer mode to a second data transfer mode, initiating storage of the first and second plurality of writes, and receiving writes provided according to one of: the first data transfer mode and the second data transfer mode, wherein a received write corresponding to data in one of the chunks is merged with the data in one of the chunks. A received write that does not correspond to data in one of the chunks may be stored according to the second data transfer mode. Initiating storage of the first and second plurality of writes may include completing storage of the first plurality of writes prior to beginning storage of the second plurality of writes. Storing data provided to a remote storage device may also include returning a first type of acknowledge message in response to receiving a write provided according to the first data transfer mode and returning a second type of acknowledge message in response to receiving a write provided according to the second data transfer mode.

According further to the present invention, computer software that handles transitioning from a first data transfer mode to a second data transfer mode that is different from the first data transfer mode includes executable code on a primary storage device that receives a plurality of data writes while being in the first data transfer mode, executable code that associates data writes begun after a first time and before a second time with a first chunk of data, executable code that associates data writes begun after the second time with a second chunk of data different from the first chunk of data, and executable code that responds to a transition to the second data transfer mode after the second time by first initiating transfer of writes associated with the first chunk of data to a secondary storage device after completion of all writes associated with the first chunk of data and then, after all writes associated with the first chunk of data have been transferred to the secondary storage device, transferring subsequent data to the secondary storage device using the first data transfer mode, wherein writes begun after initiating transitioning are provided to the secondary storage device using the second data transfer mode. The computer software may also include executable code that sends a message to the secondary storage device following the primary storage device transferring all writes associated with the first chunk of data to the secondary storage device. In the second data transfer mode, the primary storage device may acknowledge a write thereto in response to the secondary storage device acknowledging receipt of data corresponding to the write. In the second data transfer mode, the primary storage device may acknowledge a write thereto in response to the secondary storage device acknowledging receipt of data previously written to a storage location thereof corresponding to the write. The primary storage device may acknowledge the write prior to the secondary storage device acknowledging receipt of data corresponding to the write. The computer software may also include executable code that transfers writes associated with the second chunk of data to the secondary storage device following transitioning to the second data transfer mode. A subset of the writes associated with the second chunk may be transferred using the second data transfer mode. The computer software may also include executable code that inhibits creation of additional chunks of data prior to transitioning from the first data transfer mode to the second data transfer mode. The computer software may also include executable code that sends a message to the secondary storage device indicating the transitioning from the first data transfer mode to the second data transfer mode. The computer software may also include executable code that associates a data write with the second chunk of data when the data write occurs prior to transferring all writes associated with the first chunk of data to the secondary storage device. The computer software may also include executable code that merges a write with data in the second chunk of data if the write corresponds to data already in the second chunk of data when the data write occurs after transferring all writes associated with the first chunk of data to the secondary storage device and before transferring all writes associated with the second chunk of data to the secondary storage device. The computer software may also include executable code that transfers a write using the second data transfer mode if the write does not correspond to data already in the second chunk of data when the data write occurs after transferring all writes associated with the first chunk of data to the secondary storage device and before transferring all writes associated with the second chunk of data to the secondary storage device. The computer software may also include executable code that provides a transition variable that is periodically incremented to indicate a state of the transition, wherein the transition variable is used to select the first mode or the second mode for writes that occur after initiation of the transition.

According further to the present invention, computer software that stores data provided to a remote storage device includes executable code that receives a first plurality of writes associated with a first chunk of data, executable code that receives a second plurality of writes associated with a second chunk of data, wherein the second plurality of writes are all begun after the first plurality of writes, executable code that receives a message indicating a transition from a first data transfer mode to a second data transfer mode, executable code that initiates storage of the first and second plurality of writes, and executable code that receives writes provided according to one of: the first data transfer mode and the second data transfer mode, wherein a received write corresponding to data in one of the chunks is merged with the data in one of the chunks. A received write that does not correspond to data in one of the chunks may be stored according to the second data transfer mode. Executable code that initiates storage of the first and second plurality of writes may include executable code that completes storage of the first plurality of writes prior to beginning storage of the second plurality of writes. The computer software may also include executable code that returns a first type of acknowledge message in response to receiving a write provided according to the first data transfer mode and executable code that returns a second type of acknowledge message in response to receiving a write provided according to the second data transfer mode.

According further to the present invention, a data storage device includes at least one disk drive that contains data, at least one host adaptor, coupled to the at least one disk drive, that receives data to be stored on the at least one disk drive, and at least one remote adaptor, coupled to the at least one disk drive and the at least one host adaptor, that transmits data to a remote storage device, wherein, in response to receipt of data by the at least one host adaptor, data writes begun after a first time and before a second time are associated with a first chunk of data, data writes begun after the second time are associated with a second chunk of data different from the first chunk of data and, a transition from a first data transfer mode to a second data transfer mode is provided by first initiating transfer of writes associated with the first chunk of data to a secondary storage device after completion of all writes associated with the first chunk of data and then, after all writes associated with the first chunk of data have been transferred to the secondary storage device, transferring subsequent data to the secondary storage device using the first data transfer mode, wherein writes begun after initiating transitioning are provided to the secondary storage device using the second data transfer mode.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

Referring toFIG. 1, a diagram20shows a relationship between a host22, a local storage device24and a remote storage device26. The host22reads and writes data from and to the local storage device24via a host adapter (HA)28, which facilitates the interface between the host22and the local storage device24. Although the diagram20only shows one host22and one HA28, it will be appreciated by one of ordinary skill in the art that multiple HA's may be used and that one or more HA's may have one or more hosts coupled thereto.

Data from the local storage device24is copied to the remote storage device26via an RDF link29to cause the data on the remote storage device26to be identical to the data on the local storage device24. Although only the one link29is shown, it is possible to have additional links between the storage devices24,26and to have links between one or both of the storage devices24,26and other storage devices (not shown). Note that there may be a time delay between the transfer of data from the local storage device24to the remote storage device26, so that the remote storage device26may, at certain points in time, contain data that is not identical to the data on the local storage device24. Communication using RDF is described, for example, in U.S. Pat. No. 5,742,792, which is incorporated by reference herein.

The local storage device24includes a first plurality of RDF adapter units (RA's)30a,30b,30cand the remote storage device26includes a second plurality of RA's32a–32c. The RA's30a–30c,32a–32care coupled to the RDF link29and are similar to the host adapter28, but are used to transfer data between the storage devices24,26. The software used in connection with the RA's30a–30c,32a–32cis discussed in more detail hereinafter.

The storage devices24,26may include one or more disks, each containing a different portion of data stored on each of the storage devices24,26.FIG. 1shows the storage device24including a plurality of disks33a,33b,33cand the storage device26including a plurality of disks34a,34b,34c. The RDF functionality described herein may be applied so that the data for at least a portion of the disks33a–33cof the local storage device24is copied, using RDF, to at least a portion of the disks34a–34cof the remote storage device26. It is possible that other data of the storage devices24,26is not copied between the storage devices24,26, and thus is not identical.

Each of the disks33a–33cis coupled to a corresponding disk adapter unit (DA)35a,35b,35cthat provides data to a corresponding one of the disks33a–33cand receives data from a corresponding one of the disks33a–33c. Similarly, a plurality of DA's36a,36b,36cof the remote storage device26are used to provide data to corresponding ones of the disks34a–34cand receive data from corresponding ones of the disks34a–34c. An internal data path exists between the DA's35a–35c, the HA28and the RA's30a–30cof the local storage device24. Similarly, an internal data path exists between the DA's36a–36cand the RA's32a–32cof the remote storage device26. Note that, in other embodiments, it is possible for more than one disk to be serviced by a DA and that it is possible for more than one DA to service a disk.

The local storage device24also includes a global memory37that may be used to facilitate data transferred between the DA's35a–35c, the HA28and the RA's30a–30c. The memory37may contain tasks that are to be performed by one or more of the DA's35a–35c, the HA28and the RA's30a–30c, and a cache for data fetched from one or more of the disks33a–33c. Similarly, the remote storage device26includes a global memory38that may contain tasks that are to be performed by one or more of the DA's36a–36cand the RA's32a–32c, and a cache for data fetched from one or more of the disks34a–34c. Use of the memories37,38is described in more detail hereinafter.

The storage space in the local storage device24that corresponds to the disks33a–33cmay be subdivided into a plurality of volumes or logical devices. The logical devices may or may not correspond to the physical storage space of the disks33a–33c. Thus, for example, the disk33amay contain a plurality of logical devices or, alternatively, a single logical device could span both of the disks33a,33b. Similarly, the storage space for the remote storage device26that comprises the disks34a–34cmay be subdivided into a plurality of volumes or logical devices, where each of the logical devices may or may not correspond to one or more of the disks34a–34c.

Providing an RDF mapping between portions of the local storage device24and the remote storage device26involves setting up a logical device on the remote storage device26that is a remote mirror for a logical device on the local storage device24. The host22reads and writes data from and to the logical device on the local storage device24and the RDF mapping causes modified data to be transferred from the local storage device24to the remote storage device26using the RA's,30a–30c,32a–32cand the RDF link29. In steady state operation, the logical device on the remote storage device26contains data that is identical to the data of the logical device on the local storage device24. The logical device on the local storage device24that is accessed by the host22is referred to as the “R1 volume” (or just “R1”) while the logical device on the remote storage device26that contains a copy of the data on the R1 volume is called the “R2 volume” (or just “R2”). Thus, the host reads and writes data from and to the R1 volume and RDF handles automatic copying and updating of the data from the R1 volume to the R2 volume.

Referring toFIG. 2, a path of data is illustrated from the host22to the local storage device24and the remote storage device26. Data written from the host22to the local storage device24is stored locally, as illustrated by the data element51of the local storage device24. The data that is written by the host22to the local storage device24is also maintained by the local storage device24in connection with being sent by the local storage device24to the remote storage device26via the link29.

In the system described herein, each data write by the host22(of, for example a record, a plurality of records, a track, etc.) is assigned a sequence number. The sequence number may be provided in an appropriate data field associated with the write. InFIG. 2, the writes by the host22are shown as being assigned sequence number N. All of the writes performed by the host22that are assigned sequence number N are collected in a single chunk of data52. The chunk52represents a plurality of separate writes by the host22that occur at approximately the same time.

Generally, the local storage device24accumulates chunks of one sequence number while transmitting a previously accumulated chunk (having the previous sequence number) to the remote storage device26. Thus, while the local storage device24is accumulating writes from the host22that are assigned sequence number N, the writes that occurred for the previous sequence number (N−1) are transmitted by the local storage device24to the remote storage device26via the link29. A chunk54represents writes from the host22that were assigned the sequence number N−1 that have not been transmitted yet to the remote storage device26.

The remote storage device26receives the data from the chunk54corresponding to writes assigned a sequence number N−1 and constructs a new chunk56of host writes having sequence number N−1. The data may be transmitted using appropriate RDF protocol that acknowledges data sent across the link29. When the remote storage device26has received all of the data from the chunk54, the local storage device24sends a commit message to the remote storage device26to commit all the data assigned the N−1 sequence number corresponding to the chunk56. Generally, once a chunk corresponding to a particular sequence number is committed, that chunk may be written to the logical storage device. This is illustrated inFIG. 2with a chunk58corresponding to writes assigned sequence number N−2 (i.e., two before the current sequence number being used in connection with writes by the host22to the local storage device26). InFIG. 2, the chunk58is shown as being written to a data element62representing disk storage for the remote storage device26. Thus, the remote storage device26is receiving and accumulating the chunk56corresponding to sequence number N−1 while the chunk58corresponding to the previous sequence number (N−2) is being written to disk storage of the remote storage device26illustrated by the data element62. In some embodiments, the data for the chunk58is marked for write (but not necessarily written immediately), while the data for the chunk56is not.

Thus, in operation, the host22writes data to the local storage device24that is stored locally in the data element51and is accumulated in the chunk52. Once all of the data for a particular sequence number has been accumulated (described elsewhere herein), the local storage device24increments the sequence number. Data from the chunk54corresponding to one less than the current sequence number is transferred from the local storage device24to the remote storage device26via the link29. The chunk58corresponds to data for a sequence number that was committed by the local storage device24sending a message to the remote storage device26. Data from the chunk58is written to disk storage of the remote storage device26.

Note that the writes within a particular one of the chunks52,54,56,58are not necessarily ordered. However, as described in more detail elsewhere herein, every write for the chunk58corresponding to sequence number N−2 was begun prior to beginning any of the writes for the chunks54,56corresponding to sequence number N−1. In addition, every write for the chunks54,56corresponding to sequence number N−1 was begun prior to beginning any of the writes for the chunk52corresponding to sequence number N. Thus, in the event of a communication failure between the local storage device24and the remote storage device26, the remote storage device26may simply finish writing the last committed chunk of data (the chunk58in the example ofFIG. 2) and can be assured that the state of the data at the remote storage device26is ordered in the sense that the data element62contains all of the writes that were begun prior to a certain point in time and contains no writes that were begun after that point in time. Thus, R2 always contains a point in time copy of R1 and it is possible to reestablish a consistent image from the R2 device.

Referring toFIG. 3, a diagram70illustrates items used to construct and maintain the chunks52,54. A standard logical device72contains data written by the host22and corresponds to the data element51ofFIG. 2and the disks33a–33cofFIG. 1. The standard logical device72contains data written by the host22to the local storage device24.

Two linked lists of pointers74,76are used in connection with the standard logical device72. The linked lists74,76correspond to data that may be stored, for example, in the memory37of the local storage device24. The linked list74contains a plurality of pointers81–85, each of which points to a slot of a cache88used in connection with the local storage device24. Similarly, the linked list76contains a plurality of pointers91–95, each of which points to a slot of the cache88. In some embodiments, the cache88may be provided in the memory37of the local storage device24. The cache88contains a plurality of cache slots102–104that may be used in connection to writes to the standard logical device72and, at the same time, used in connection with the linked lists74,76.

Each of the linked lists74,76may be used for one of the chunks of data52,54so that, for example, the linked list74may correspond to the chunk of data52for sequence number N while the linked list76may correspond to the chunk of data54for sequence number N−1. Thus, when data is written by the host22to the local storage device24, the data is provided to the cache88and, in some cases (described elsewhere herein), an appropriate pointer of the linked list74is created. Note that the data will not be removed from the cache88until the data is destaged to the standard logical device72and the data is also no longer pointed to by one of the pointers81–85of the linked list74, as described elsewhere herein.

In an embodiment herein, one of the linked lists74,76is deemed “active” while the other is deemed “inactive”. Thus, for example, when the sequence number N is even, the linked list74may be active while the linked list76is inactive. The active one of the linked lists74,76handles writes from the host22while the inactive one of the linked lists74,76corresponds to the data that is being transmitted from the local storage device24to the remote storage device26.

While the data that is written by the host22is accumulated using the active one of the linked lists74,76(for the sequence number N), the data corresponding to the inactive one of the linked lists74,76(for previous sequence number N−1) is transmitted from the local storage device24to the remote storage device26. The RA's30a–30cuse the linked lists74,76to determine the data to transmit from the local storage device24to the remote storage device26.

Once data corresponding to a particular one of the pointers in one of the linked lists74,76has been transmitted to the remote storage device26, the particular one of the pointers may be removed from the appropriate one of the linked lists74,76. In addition, the data may also be marked for removal from the cache88(i.e., the slot may be returned to a pool of slots for later, unrelated, use) provided that the data in the slot is not otherwise needed for another purpose (e.g., to be destaged to the standard logical device72). A mechanism may be used to ensure that data is not removed from the cache88until all devices are no longer using the data. Such a mechanism is described, for example, in U.S. Pat. No. 5,537,568 issued on Jul. 16, 1996 and in U.S. patent application Ser. No. 09/850,551 filed on Jul. 7, 2001, both of which are incorporated by reference herein.

Referring toFIG. 4, a slot120, like one of the slots102–104of the cache88, includes a header122and data124. The header122corresponds to overhead information used by the system to manage the slot120. The data124is the corresponding data from the disk that is being (temporarily) stored in the slot120. Information in the header122includes pointers back to the disk, time stamp(s), etc.

The header122also includes a cache stamp126used in connection with the system described herein. In an embodiment herein, the cache stamp126is eight bytes. Two of the bytes are a “password” that indicates whether the slot120is being used by the system described herein. In other embodiments, the password may be one byte while the following byte is used for a pad. As described elsewhere herein, the two bytes of the password (or one byte, as the case may be) being equal to a particular value indicates that the slot120is pointed to by at least one entry of the linked lists74,76. The password not being equal to the particular value indicates that the slot120is not pointed to by an entry of the linked lists74,76. Use of the password is described elsewhere herein.

The cache stamp126also includes a two byte field indicating the sequence number (e.g., N, N−1, N−2, etc.) of the data124of the slot120. As described elsewhere herein, the sequence number field of the cache stamp126may be used to facilitate the processing described herein. The remaining four bytes of the cache stamp126may be used for a pointer, as described elsewhere herein. Of course, the two bytes of the sequence number and the four bytes of the pointer are only valid when the password equals the particular value that indicates that the slot120is pointed to by at least one entry in one of the lists74,76.

Referring toFIG. 5, a flow chart140illustrates steps performed by the HA28in connection with a host22performing a write operation. Of course, when the host22performs a write, processing occurs for handling the write in a normal fashion irrespective of whether the data is part of an R1/R2 RDF group. For example, when the host22writes data for a portion of the disk, the write occurs to a cache slot which is eventually destaged to the disk. The cache slot may either be a new cache slot or may be an already existing cache slot created in connection with a previous read and/or write operation to the same track.

Processing begins at a first step142where a slot corresponding to the write is locked. In an embodiment herein, each of the slots102–104of the cache88corresponds to a track of data on the standard logical device72. Locking the slot at the step142prevents additional processes from operating on the relevant slot during the processing performed by the HA28corresponding to the steps of the flow chart140.

Following step142is a step144where a value for N, the sequence number, is set. As discussed elsewhere herein, the value for the sequence number obtained at the step144is maintained during the entire write operation performed by the HA28while the slot is locked. As discussed elsewhere herein, the sequence number is assigned to each write to set the one of the chunks of data52,54to which the write belongs. Writes performed by the host22are assigned the current sequence number. It is useful that a single write operation maintain the same sequence number throughout.

Following the step144is a test step146which determines if the password field of the cache slot is valid. As discussed above, the system described herein sets the password field to a predetermined value to indicate that the cache slot is already in one of the linked lists of pointers74,76. If it is determined at the test step146that the password field is not valid (indicating that the slot is new and that no pointers from the lists74,76point to the slot), then control passes from the step146to a step148, where the cache stamp of the new slot is set by setting the password to the predetermined value, setting the sequence number field to N, and setting the pointer field to Null. In other embodiments, the pointer field may be set to point to the slot itself.

Following the step148is a step152where a pointer to the new slot is added to the active one of the pointer lists74,76. In an embodiment herein, the lists74,76are circular doubly linked lists, and the new pointer is added to the circular doubly linked list in a conventional fashion. Of course, other appropriate data structures could be used to manage the lists74,76. Following the step152is a step154where flags are set. At the step154, the RDF_WP flag (RDF write pending flag) is set to indicate that the slot needs to be transmitted to the remote storage device26using RDF. In addition, at the step154, the IN_CACHE flag is set to indicate that the slot needs to be destaged to the standard logical device72. Following the step154is a step156where the data being written by the host22and the HA28is written to the slot. Following the step156is a step158where the slot is unlocked. Following step158, processing is complete.

If it is determined at the test step146that the password field of the slot is valid (indicating that the slot is already pointed to by at least one pointer of the lists74,76), then control transfers from the step146to a test step162, where it is determined whether the sequence number field of the slot is equal to the current sequence number, N. Note that there are two valid possibilities for the sequence number field of a slot with a valid password. It is possible for the sequence number field to be equal to N, the current sequence number. This occurs when the slot corresponds to a previous write with sequence number N. The other possibility is for the sequence number field to equal N−1. This occurs when the slot corresponds to a previous write with sequence number N−1. Any other value for the sequence number field is invalid. Thus, for some embodiments, it may be possible to include error/validity checking in the step162or possibly make error/validity checking a separate step. Such an error may be handled in any appropriate fashion, which may include providing a message to a user.

If it is determined at the step162that the value in the sequence number field of the slot equals the current sequence number N, then no special processing is required and control transfers from the step162to the step156, discussed above, where the data is written to the slot. Otherwise, if the value of the sequence number field is N−1 (the only other valid value), then control transfers from the step162to a step164where a new slot is obtained. The new slot obtained at the step164may be used to store the data being written.

Following the step164is a step166where the data from the old slot is copied to the new slot that was obtained at the step164. Note that that the copied data includes the RDF_WP flag, which should have been set at the step154on a previous write when the slot was first created. Following the step166is a step168where the cache stamp for the new slot is set by setting the password field to the appropriate value, setting the sequence number field to the current sequence number, N, and setting the pointer field to point to the old slot. Following the step168is a step172where a pointer to the new slot is added to the active one of the linked lists74,76. Following the step172is the step156, discussed above, where the data is written to the slot which, in this case, is the new slot.

Referring toFIG. 6, a flow chart200illustrates steps performed in connection with the RA's30a–30cscanning the inactive one of the lists72,74to transmit RDF data from the local storage device24to the remote storage device26. As discussed above, the inactive one of the lists72,74points to slots corresponding to the N−1 cycle for the R1device when the N cycle is being written to the R1device by the host using the active one of the lists72,74.

Processing begins at a first step202where it is determined if there are any entries in the inactive one of the lists72,74. As data is transmitted, the corresponding entries are removed from the inactive one of the lists72,74. In addition, new writes are provided to the active one of the lists72,74and not generally to the inactive one of the lists72,74. Thus, it is possible (and desirable, as described elsewhere herein) for the inactive one of the lists72,74to contain no data at certain times. If it is determined at the step202that there is no data to be transmitted, then the inactive one of the lists72,74is continuously polled until data becomes available. Data for sending becomes available in connection with a cycle switch (discussed elsewhere herein) where the inactive one of the lists72,74becomes the active one of the lists72,74, and vice versa.

If it is determined at the step202that there is data available for sending, control transfers from the step202to a step204, where the slot is verified as being correct. The processing performed at the step204is an optional “sanity check” that may include verifying that the password field is correct and verifying that the sequence number field is correct. If there is incorrect (unexpected) data in the slot, error processing may be performed, which may include notifying a user of the error and possibly error recovery processing.

Following the step204is a step212, where the data is sent via RDF in a conventional fashion. In an embodiment herein, the entire slot is not transmitted. Rather, only records within the slot that have the appropriate mirror bits set (indicating the records have changed) are transmitted to the remote storage device26. However, in other embodiments, it may be possible to transmit the entire slot, provided that the remote storage device26only writes data corresponding to records having appropriate mirror bits set and ignores other data for the track, which may or may not be valid. Following the step212is a test step214where it is determined if the data that was transmitted has been acknowledged by the R2device. If not, the data is resent, as indicated by the flow from the step214back to the step212. In other embodiments, different and more involved processing may used to send data and acknowledge receipt thereof. Such processing may include error reporting and alternative processing that is performed after a certain number of attempts to send the data have failed.

Once it is determined at the test step214that the data has been successfully sent, control passes from the step214to a step216to clear the RDF_WP flag (since the data has been successfully sent via RDF). Following the step216is a test step218where it is determined if the slot is a duplicate slot created in connection with a write to a slot already having an existing entry in the inactive one of the lists72,74. This possibility is discussed above in connection with the steps162,164,166,168,172. If it is determined at the step218that the slot is a duplicate slot, then control passes from the step218to a step222where the slot is returned to the pool of available slots (to be reused). In addition, the slot may also be aged (or have some other appropriate mechanism applied thereto) to provide for immediate reuse ahead of other slots since the data provided in the slot is not valid for any other purpose. Following the step222or the step218if the slot is not a duplicate slot is a step224where the password field of the slot header is cleared so that when the slot is reused, the test at the step146ofFIG. 5properly classifies the slot as a new slot.

Following the step224is a step226where the entry in the inactive one of the lists72,74is removed. Following the step226, control transfers back to the step202, discussed above, where it is determined if there are additional entries on the inactive one of the lists72,74corresponding to data needing to be transferred.

Referring toFIG. 7, a diagram240illustrates creation and manipulation of the chunks56,58used by the remote storage device26. Data that is received by the remote storage device26, via the link29, is provided to a cache242of the remote storage device26. The cache242may be provided, for example, in the memory38of the remote storage device26. The cache242includes a plurality of cache slots244–246, each of which may be mapped to a track of a standard logical storage device252. The cache242is similar to the cache88ofFIG. 3and may contain data that can be destaged to the standard logical storage device252of the remote storage device26. The standard logical storage device252corresponds to the data element62shown inFIG. 2and the disks34a–34cshown inFIG. 1.

The remote storage device26also contains a pair of cache only virtual devices254,256. The cache only virtual devices254,256corresponded device tables that may be stored, for example, in the memory38of the remote storage device26. Each track entry of the tables of each of the cache only virtual devices254,256point to either a track of the standard logical device252or point to a slot of the cache242. Cache only virtual devices are described in a copending U.S. patent application titled CACHE-ONLY VIRTUAL DEVICES, filed on Mar. 25, 2003 and having Ser. No. 10/396,800, which is incorporated by reference herein.

The plurality of cache slots244–246may be used in connection to writes to the standard logical device252and, at the same time, used in connection with the cache only virtual devices254,256. In an embodiment herein, each of track table entry of the cache only virtual devices254,256contain a null to indicate that the data for that track is stored on a corresponding track of the standard logical device252. Otherwise, an entry in the track table for each of the cache only virtual devices254,256contains a pointer to one of the slots244–246in the cache242.

Each of the cache only virtual devices254,256corresponds to one of the data chunks56,58. Thus, for example, the cache only virtual device254may correspond to the data chunk56while the cache only virtual device256may correspond to the data chunk58. In an embodiment herein, one of the cache only virtual devices254,256may be deemed “active” while the other one of the cache only virtual devices254,256may be deemed “inactive”. The inactive one of the cache only virtual devices254,256may correspond to data being received from the local storage device24(i.e., the chunk56) while the active one of the cache only virtual device254,256corresponds to data being restored (written) to the standard logical device252.

Data from the local storage device24that is received via the link29may be placed in one of the slots244–246of the cache242. A corresponding pointer of the inactive one of the cache only virtual devices254,256may be set to point to the received data. Subsequent data having the same sequence number may be processed in a similar manner. At some point, the local storage device24provides a message committing all of the data sent using the same sequence number. Once the data for a particular sequence number has been committed, the inactive one of the cache only virtual devices254,256becomes active and vice versa. At that point, data from the now active one of the cache only virtual devices254,256is copied to the standard logical device252while the inactive one of the cache only virtual devices254,256is used to receive new data (having a new sequence number) transmitted from the local storage device24to the remote storage device26.

As data is removed from the active one of the cache only virtual devices254,256(discussed elsewhere herein), the corresponding entry in the active one of the cache only virtual devices254,256may be set to null. In addition, the data may also be removed from the cache244(i.e., the slot returned to the pool of free slots for later use) provided that the data in the slot is not otherwise needed for another purpose (e.g., to be destaged to the standard logical device252). A mechanism may be used to ensure that data is not removed from the cache242until all mirrors (including the cache only virtual devices254,256) are no longer using the data. Such a mechanism is described, for example, in U.S. Pat. No. 5,537,568 issued on Jul. 16, 1996 and in U.S. patent application Ser. No. 09/850,551 filed on Jul. 7, 2001, both of which are incorporated by reference herein.

In some embodiments discussed elsewhere herein, the remote storage device26may maintain linked lists258,262like the lists74,76used by the local storage device24. The lists258,262may contain information that identifies the slots of the corresponding cache only virtual devices254,256that have been modified, where one of the lists258,262corresponds to one of the cache only virtual devices254,256and the other one of the lists258,262corresponds to the other one of the cache only virtual devices254,256. As discussed elsewhere herein, the lists258,262may be used to facilitate restoring data from the cache only virtual devices254,256to the standard logical device252.

Referring toFIG. 8, a flow chart270illustrates steps performed by the remote storage device26in connection with processing data for a sequence number commit transmitted by the local storage device24to the remote storage device26. As discussed elsewhere herein, the local storage device24periodically increments sequence numbers. When this occurs, the local storage device24finishes transmitting all of the data for the previous sequence number and then sends a commit message for the previous sequence number.

Processing begins at a first step272where the commit is received. Following the step272is a test step274which determines if the active one of the cache only virtual devices254,256of the remote storage device26is empty. As discussed elsewhere herein, the inactive one of the cache only virtual devices254,256of the remote storage device26is used to accumulate data from the local storage device24sent using RDF while the active one of the cache only virtual devices254,256is restored to the standard logical device252.

If it is determined at the test step274that the active one of the cache only virtual devices254,256is not empty, then control transfers from the test step274to a step276where the restore for the active one of the cache only virtual devices254,256is completed prior to further processing being performed. Restoring data from the active one of the cache only virtual devices254,256is described in more detail elsewhere herein. It is useful that the active one of the cache only virtual devices254,256is empty prior to handling the commit and beginning to restore data for the next sequence number.

Following the step276or following the step274if the active one of the cache only virtual devices254,256is determined to be empty, is a step278where the active one of the cache only virtual devices254,256is made inactive. Following the step278is a step282where the previously inactive one of the cache only virtual devices254,256(i.e., the one that was inactive prior to execution of the step278) is made active. Swapping the active and inactive cache only virtual devices254,256at the steps278,282prepares the now inactive (and empty) one of the cache only virtual devices254,256to begin to receive data from the local storage device24for the next sequence number.

Following the step282is a step284where the active one of the cache only virtual devices254,256is restored to the standard logical device252of the remote storage device26. Restoring the active one of the cache only virtual devices254,256to the standard logical device252is described in more detail hereinafter. However, note that, in some embodiments, the restore process is begun, but not necessarily completed, at the step284. Following the step284is a step286where the commit that was sent from the local storage device24to the remote storage device26is acknowledged back to the local storage device24so that the local storage device24is informed that the commit was successful. Following the step286, processing is complete.

Referring toFIG. 9, a flow chart300illustrates in more detail the steps276,284ofFIG. 8where the remote storage device26restores the active one of the cache only virtual devices254,256. Processing begins at a first step302where a pointer is set to point to the first slot of the active one of the cache only virtual devices254,256. The pointer is used to iterate through each track table entry of the active one of the cache only virtual devices254,256, each of which is processed individually. Following the step302is a test step304where it is determined if the track of the active one of the cache only virtual devices254,256that is being processed points to the standard logical device252. If so, then there is nothing to restore. Otherwise, control transfers from the step304to a step a306where the corresponding slot of the active one of the cache only virtual devices254,256is locked.

Following the step306is a test step308which determines if the corresponding slot of the standard logical device252is already in the cache of the remote storage device26. If so, then control transfers from the test step308to a step312where the slot of the standard logical device is locked. Following step312is a step314where the data from the active one of the cache only virtual devices254,256is merged with the data in the cache for the standard logical device252. Merging the data at the step314involves overwriting the data for the standard logical device with the new data of the active one of the cache only virtual devices254,256. Note that, in embodiments that provide for record level flags, it may be possible to simply OR the new records from the active one of the cache only virtual devices254,256to the records of the standard logical device252in the cache. That is, if the records are interleaved, then it is only necessary to use the records from the active one of the cache only virtual devices254,256that have changed and provide the records to the cache slot of the standard logical device252. Following step314is a step316where the slot of the standard logical device252is unlocked. Following step316is a step318where the slot of the active one of the cache only virtual devices254,256that is being processed is also unlocked.

If it is determined at the test step308that the corresponding slot of the standard logical device252is not in cache, then control transfers from the test step308to a step322where the track entry for the slot of the standard logical device252is changed to indicate that the slot of the standard logical device252is in cache (e.g., an IN_CACHE flag may be set) and needs to be destaged. As discussed elsewhere herein, in some embodiments, only records of the track having appropriate mirror bits set may need to be destaged. Following the step322is a step324where a flag for the track may be set to indicate that the data for the track is in the cache.

Following the step324is a step326where the slot pointer for the standard logical device252is changed to point to the slot in the cache. Following the step326is a test step328which determines if the operations performed at the steps322,324,326have been successful. In some instances, a single operation called a “compare and swap” operation may be used to perform the steps322,324,326. If these operations are not successful for any reason, then control transfers from the step328back to the step308to reexamine if the corresponding track of the standard logical device252is in the cache. Otherwise, if it is determined at the test step328that the previous operations have been successful, then control transfers from the test step328to the step318, discussed above.

Following the step318is a test step332which determines if the cache slot of the active one of the cache only virtual devices254,256(which is being restored) is still being used. In some cases, it is possible that the slot for the active one of the cache only virtual devices254,256is still being used by another mirror. If it is determined at the test step332that the slot of the cache only virtual device is not being used by another mirror, then control transfers from the test step332to a step334where the slot is released for use by other processes (e.g., restored to pool of available slots, as discussed elsewhere herein). Following the step334is a step336to point to the next slot to process the next slot of the active one of the cache only virtual devices254,256. Note that the step336is also reached from the test step332if it is determined at the step332that the active one of the cache only virtual devices254,256is still being used by another mirror. Note also that the step336is reached from the test step304if it is determined at the step304that, for the slot being processed, the active one of the cache only virtual devices254,256points to the standard logical device252. Following the step336is a test step338which determines if there are more slots of the active one of the cache only virtual devices254,256to be processed. If not, processing is complete. Otherwise, control transfers from the test step338back to the step304.

In another embodiment, it is possible to construct lists of modified slots for the received chunk of data56corresponding to the N−1 cycle on the remote storage device26, such as the lists258,262shown inFIG. 7. As the data is received, the remote storage device26constructs a linked list of modified slots. The lists that are constructed may be circular, linear (with a NULL termination), or any other appropriate design. The lists may then be used to restore the active one of the cache only virtual devices254,256.

The flow chart300ofFIG. 9shows two alternative paths342,344that illustrate operation of embodiments where a list of modified slots is used. At the step302, a pointer (used for iterating through the list of modified slots) is made to point to the first element of the list. Following the step302is the step306, which is reached by the alternative path342. In embodiments that use lists of modified slots, the test step304is not needed since no slots on the list should point to the standard logical device252.

Following the step306, processing continues as discussed above with the previous embodiment, except that the step336refers to traversing the list of modified slots rather than pointing to the next slot in the COVD. Similarly, the test at the step338determines if the pointer is at the end of the list (or back to the beginning in the case of a circular linked list). Also, if it is determined at the step338that there are more slots to process, then control transfers from the step338to the step306, as illustrated by the alternative path344. As discussed above, for embodiments that use a list of modified slots, the step304may be eliminated.

Referring toFIG. 10, a flow chart350illustrates steps performed in connection with the local storage device24increasing the sequence number. Processing begins at a first step352where the local storage device24waits at least M seconds prior to increasing the sequence number. In an embodiment herein, M is thirty, but of course M could be any number. Larger values for M increase the amount of data that may be lost if communication between the storage devices24,26is disrupted. However, smaller values for M increase the total amount of overhead caused by incrementing the sequence number more frequently.

Following the step352is a test step354which determines if all of the HA's of the local storage device24have set a bit indicating that the HA's have completed all of the I/O's for a previous sequence number. When the sequence number changes, each of the HA's notices the change and sets a bit indicating that all I/O's of the previous sequence number are completed. For example, if the sequence number changes from N−1 to N, an HA will set the bit when the HA has completed all I/O's for sequence number N−1. Note that, in some instances, a single I/O for an HA may take a long time and may still be in progress even after the sequence number has changed. Note also that, for some systems, a different mechanism may be used to determine if all of the HA's have completed their N−1 I/O's. The different mechanism may include examining device tables in the memory37.

If it is determined at the test step354that I/O's from the previous sequence number have been completed, then control transfers from the step354to a test step356which determines if the inactive one of the lists74,76is empty. Note that a sequence number switch may not be made unless and until all of the data corresponding to the inactive one of the lists74,76has been completely transmitted from the local storage device24to the remote storage device26using the RDF protocol. Once the inactive one of the lists74,76is determined to be empty, then control transfers from the step356to a step358where the commit for the previous sequence number is sent from the local storage device24to the remote storage device26. As discussed above, the remote storage device26receiving a commit message for a particular sequence number will cause the remote storage device26to begin restoring the data corresponding to the sequence number.

Following the step358is a step362where the copying of data for the inactive one of the lists74,76is suspended. As discussed elsewhere herein, the inactive one of the lists is scanned to send corresponding data from the local storage device24to the remote storage device26. It is useful to suspend copying data until the sequence number switch is completed. In an embodiment herein, the suspension is provided by sending a message to the RA's30a–30c. However, it will be appreciated by one of ordinary skill in the art that for embodiments that use other components to facilitate sending data using the system described herein, suspending copying may be provided by sending appropriate messages/commands to the other components.

Following step362is a step364where the sequence number is incremented. Following step364is a step366where the bits for the HA's that are used in the test step354are all cleared so that the bits may be set again in connection with the increment of the sequence number. Following step366is a test step372which determines if the remote storage device26has acknowledged the commit message sent at the step358. Acknowledging the commit message is discussed above in connection withFIG. 8. Once it is determined that the remote storage device26has acknowledged the commit message sent at the step358, control transfers from the step372to a step374where the suspension of copying, which was provided at the step362, is cleared so that copying may resume. Following step374, processing is complete. Note that it is possible to go from the step374back to the step352to begin a new cycle to continuously increment the sequence number.

It is also possible to use COVD's on the R1device to collect slots associated with active data and inactive chunks of data. In that case, just as with the R2device, one COVD could be associated with the inactive sequence number and another COVD could be associated with the active sequence number. This is described below.

Referring toFIG. 11, a diagram400illustrates items used to construct and maintain the chunks52,54. A standard logical device402contains data written by the host22and corresponds to the data element51ofFIG. 2and the disks33a–33cofFIG. 1. The standard logical device402contains data written by the host22to the local storage device24.

Two cache only virtual devices404,406are used in connection with the standard logical device402. The cache only virtual devices404,406corresponded device tables that may be stored, for example, in the memory37of the local storage device24. Each track entry of the tables of each of the cache only virtual devices404,406point to either a track of the standard logical device402or point to a slot of a cache408used in connection with the local storage device24. In some embodiments, the cache408may be provided in the memory37of the local storage device24.

The cache408contains a plurality of cache slots412–414that may be used in connection to writes to the standard logical device402and, at the same time, used in connection with the cache only virtual devices404,406. In an embodiment herein, each track table entry of the cache only virtual devices404,406contains a null to point to a corresponding track of the standard logical device402. Otherwise, an entry in the track table for each of the cache only virtual devices404,406contains a pointer to one of the slots412–414in the cache408.

Each of the cache only virtual devices404,406may be used for one of the chunks of data52,54so that, for example, the cache only virtual device404may correspond to the chunk of data52for sequence number N while the cache only virtual device406may correspond to the chunk of data54for sequence number N−1. Thus, when data is written by the host22to the local storage device24, the data is provided to the cache408and an appropriate pointer of the cache only virtual device404is adjusted. Note that the data will not be removed from the cache408until the data is destaged to the standard logical device402and the data is also released by the cache only virtual device404, as described elsewhere herein.

In an embodiment herein, one of the cache only virtual devices404,406is deemed “active” while the other is deemed “inactive”. Thus, for example, when the sequence number N is even, the cache only virtual device404may be active while the cache only virtual device406is inactive. The active one of the cache only virtual devices404,406handles writes from the host22while the inactive one of the cache only virtual devices404,406corresponds to the data that is being transmitted from the local storage device24to the remote storage device26.

While the data that is written by the host22is accumulated using the active one of the cache only virtual devices404,406(for the sequence number N), the data corresponding to the inactive one of the cache only virtual devices404,406(for previous sequence number N−1) is transmitted from the local storage device24to the remote storage device26. For this and related embodiments, the DA's35a–35cof the local storage device handle scanning the inactive one of the cache only virtual devices404,406to send copy requests to one or more of the RA's30a–30cto transmit the data from the local storage device24to the remote storage device26. Thus, the steps362,374, discussed above in connection with suspending and resuming copying, may include providing messages/commands to the DA's35a–35c.

Once the data has been transmitted to the remote storage device26, the corresponding entry in the inactive one of the cache only virtual devices404,406may be set to null. In addition, the data may also be removed from the cache408(i.e., the slot returned to the pool of slots for later use) if the data in the slot is not otherwise needed for another purpose (e.g., to be destaged to the standard logical device402). A mechanism may be used to ensure that data is not removed from the cache408until all mirrors (including the cache only virtual devices404,406) are no longer using the data. Such a mechanism is described, for example, in U.S. Pat. No. 5,537,568 issued on Jul. 16, 1996 and in U.S. patent application Ser. No. 09/850,551 filed on Jul. 7, 2001, both of which are incorporated by reference herein.

Referring toFIG. 12, a flow chart440illustrates steps performed by the HA28in connection with a host22performing a write operation for embodiments where two COVD's are used by the R1device to provide the system described herein. Processing begins at a first step442where a slot corresponding to the write is locked. In an embodiment herein, each of the slots412–414of the cache408corresponds to a track of data on the standard logical device402. Locking the slot at the step442prevents additional processes from operating on the relevant slot during the processing performed by the HA28corresponding to the steps of the flow chart440.

Following the step442is a step444where a value for N, the sequence number, is set. Just as with the embodiment that uses lists rather than COVD's on the R1side, the value for the sequence number obtained at the step444is maintained during the entire write operation performed by the HA28while the slot is locked. As discussed elsewhere herein, the sequence number is assigned to each write to set the one of the chunks of data52,54to which the write belongs. Writes performed by the host22are assigned the current sequence number. It is useful that a single write operation maintain the same sequence number throughout.

Following the step444is a test step446, which determines if the inactive one of the cache only virtual devices404,406already points to the slot that was locked at the step442(the slot being operated upon). This may occur if a write to the same slot was provided when the sequence number was one less than the current sequence number. The data corresponding to the write for the previous sequence number may not yet have been transmitted to the remote storage device26.

If it is determined at the test step446that the inactive one of the cache only virtual devices404,406does not point to the slot, then control transfers from the test step446to another test step448, where it is determined if the active one of the cache only virtual devices404,406points to the slot. It is possible for the active one of the cache only virtual devices404,406to point to the slot if there had been a previous write to the slot while the sequence number was the same as the current sequence number. If it is determined at the test step448that the active one of the cache only virtual devices404,406does not point to the slot, then control transfers from the test step448to a step452where a new slot is obtained for the data. Following the step452is a step454where the active one of the cache only virtual devices404,406is made to point to the slot.

Following the step454, or following the step448if the active one of the cache only virtual devices404,406points to the slot, is a step456where flags are set. At the step456, the RDF_WP flag (RDF write pending flag) is set to indicate that the slot needs to be transmitted to the remote storage device26using RDF. In addition, at the step456, the IN_CACHE flag is set to indicate that the slot needs to be destaged to the standard logical device402. Note that, in some instances, if the active one of the cache only virtual devices404,406already points to the slot (as determined at the step448) it is possible that the RDF_WP and IN_CACHE flags were already set prior to execution of the step456. However, setting the flags at the step456ensures that the flags are set properly no matter what the previous state.

Following the step456is a step458where an indirect flag in the track table that points to the slot is cleared, indicating that the relevant data is provided in the slot and not in a different slot indirectly pointed to. Following the step458is a step462where the data being written by the host22and the HA28is written to the slot. Following the step462is a step464where the slot is unlocked. Following step464, processing is complete.

If it is determined at the test step446that the inactive one of the cache only virtual devices404,406points to the slot, then control transfers from the step446to a step472, where a new slot is obtained. The new slot obtained at the step472may be used for the inactive one of the cache only virtual devices404,406to effect the RDF transfer while the old slot may be associated with the active one of the cache only virtual devices404,406, as described below.

Following the step472is a step474where the data from the old slot is copied to the new slot that was obtained at the step472. Following the step474is a step476where the indirect flag (discussed above) is set to indicate that the track table entry for the inactive one of the cache only virtual devices404,406points to the old slot but that the data is in the new slot which is pointed to by the old slot. Thus, setting indirect flag at the step476affects the track table of the inactive one of the cache only virtual devices404,406to cause the track table entry to indicate that the data is in the new slot.

Following the step476is a step478where the mirror bits for the records in the new slot are adjusted. Any local mirror bits that were copied when the data was copied from the old slot to the new slot at the step474are cleared since the purpose of the new slot is to simply effect the RDF transfer for the inactive one of the cache only virtual devices. The old slot will be used to handle any local mirrors. Following the step478is the step462where the data is written to the slot. Following step462is the step464where the slot is unlocked. Following the step464, processing is complete.

Referring toFIG. 13, a flow chart500illustrates steps performed in connection with the local storage device24transmitting the chunk of data54to the remote storage device26. The transmission essentially involves scanning the inactive one of the cache only virtual devices404,406for tracks that have been written thereto during a previous iteration when the inactive one of the cache only virtual devices404,406was active. In this embodiment, the DA's35a–35cof the local storage device24scan the inactive one of the cache only virtual devices404,406to copy the data for transmission to the remote storage device26by one or more of the RA's30a–30cusing the RDF protocol.

Processing begins at a first step502where the first track of the inactive one of the cache only virtual devices404,406is pointed to in order to begin the process of iterating through all of the tracks. Following the first step502is a test step504where it is determined if the RDF_WP flag is set. As discussed elsewhere herein, the RDF_WP flag is used to indicate that a slot (track) contains data that needs to be transmitted via the RDF link. The RDF_WP flag being set indicates that at least some data for the slot (track) is to be transmitted using RDF. In an embodiment herein, the entire slot is not transmitted. Rather, only records within the slot that have the appropriate mirror bits set (indicating the records have changed) are transmitted to the remote storage device26. However, in other embodiments, it may be possible to transmit the entire slot, provided that the remote storage device26only writes data corresponding to records having appropriate mirror bits set and ignores other data for the track, which may or may not be valid.

If it is determined at the test step504that the cache slot being processed has the RDF_WP flag set, then control transfers from the step504to a test step505, where it is determined if the slot contains the data or if the slot is an indirect slot that points to another slot that contains the relevant data. In some instances, a slot may not contain the data for the portion of the disk that corresponds to the slot. Instead, the slot may be an indirect slot that points to another slot that contains the data. If it is determined at the step505that the slot is an indirect slot, then control transfers from the step505to a step506, where the data (from the slot pointed to by the indirect slot) is obtained. Thus, if the slot is a direct slot, the data for being sent by RDF is stored in the slot while if the slot is an indirect slot, the data for being sent by RDF is in another slot pointed to by the indirect slot.

Following the step506or the step505if the slot is a direct slot is a step507where data being sent (directly or indirectly from the slot) is copied by one of the DA's35a–35cto be sent from the local storage device24to the remote storage device26using the RDF protocol. Following the step507is a test step508where it is determined if the remote storage device26has acknowledged receipt of the data. If not, then control transfers from the step508back to the step507to resend the data. In other embodiments, different and more involved processing may used to send data and acknowledge receipt thereof. Such processing may include error reporting and alternative processing that is performed after a certain number of attempts to send the data have failed.

Once it is determined at the test step508that the data has been successfully sent, control passes from the step508to a step512to clear the RDF_WP flag (since the data has been successfully sent via RDF). Following the step512is a step514where appropriate mirror flags are cleared to indicate that at least the RDF mirror (R2) no longer needs the data. In an embodiment herein, each record that is part of a slot (track) has individual mirror flags indicating which mirrors use the particular record. The R2device is one of the mirrors for each of the records and it is the flags corresponding to the R2device that are cleared at the step514.

Following the step514is a test step516which determines if any of the records of the track being processed have any other mirror flags set (for other mirror devices). If not, then control passes from the step516to a step518where the slot is released (i.e., no longer being used). In some embodiments, unused slots are maintained in a pool of slots available for use. Note that if additional flags are still set for some of the records of the slot, it may mean that the records need to be destaged to the standard logical device402or are being used by some other mirror (including another R2device). Following the step518, or following the step516if more mirror flags are present, is a step522where the pointer that is used to iterate through each track entry of the inactive one of the cache only virtual devices404,406is made to point to the next track. Following the step522is a test step524which determines if there are more tracks of the inactive one of the cache only virtual devices404,406to be processed. If not, then processing is complete. Otherwise, control transfers back to the test step504, discussed above. Note that the step522is also reached from the test step504if it is determined that the RDF_WP flag is not set for the track being processed.

In some instances, it may be desirable to switch operation of the system from using virtual ordered writes with chunks of data as described herein (virtual ordered writes mode) to using synchronous RDF transfer mode (J0or Journal zero mode) or semi-synchronous RDF transfer mode (J1or Journal one mode). The choice between using the virtual ordered writes mode as described herein and using synchronous or semi-synchronous RDF transfer mode depends on a number of factors, including tolerance for delay, desired throughput, and tolerance for loss of data. It is useful if, during the transition, the integrity of the data transfer be maintained so that even if there is, for example, a failure of the local storage device or in the communication channel during the transition from virtual ordered writes mode to J0/J1RDF transfer mode, the data at the remote storage device will be consistent so that recovery may be performed at the remote storage device.

As set forth in more detail elsewhere herein, the transition is provided by emptying the N−1 chunk of data54at the local storage device24and then sending most subsequent data using J0(or J1) RDF transfer mode while emptying the other chunks of data52,56,58. Note also that, for the discussion that follows, descriptions involving transitioning to the J0RDF transfer mode applies also to transition to the J1RDF transfer mode (or other appropriate RDF modes) and vice versa.

Referring toFIG. 14, a flow chart540illustrates steps performed in connection with the local storage device24transitioning from virtual ordered writes mode to synchronous or semi-synchronous RDF transfer mode. The local storage device24may switch in response to a command from the host22or from some other entity. In some embodiments, the local storage device24may monitor data transfers and use particular metrics to automatically determine that a switch is appropriate.

Processing for the flow chart540begins at a first step542where cycle changes are frozen at the local storage device24. As discussed elsewhere herein, the cycle number, N, is used in connection with transferring data in the virtual ordered writes mode. After execution of the step542, there are no more changes in the cycle number N used by the virtual ordered writes system. Following the step542is a step544where a message is sent to the remote storage device26so that the remote storage device26may perform appropriate processing for the transition to synchronous or semi-synchronous RDF transfer mode, as described below.

Following the step544is a test step546which determines if the N−1 chunk of data54at the local storage device24is empty. If not, then control loops back to the test step546to continue to poll until the N−1 chunk of data54is empty. The local storage device24cannot perform any writes in synchronous or semi-synchronous RDF transfer mode until the N−1 chunk of data54on the local storage device24is empty. Following the step546, once the N−1 chunk of data54on the local storage device24is empty, is a step548where the local storage device24begins transmitting data from the N chunk of data52from the local storage device24to the remote storage device26. The data from the N chunk of data52is transmitted using the same protocol that is used for sending the inactive data during steady state operation of the virtual ordered writes mode. Following the step548, processing is complete. At the end of the process illustrated by the flow chart540, the chunks of data52,54of the local storage device24should be empty or should be in the process of being emptied.

Referring toFIG. 15, a flow chart560illustrates steps performed by the remote storage device26in response to receiving the message from the local storage device24sent at the step544to indicate that a switch to synchronous or semi-synchronous RDF transfer mode is taking place. Processing begins at a first step562where the data from the N−2 chunk of data58at the remote storage device26is written to the disk62. Note that the step562may not be necessary since the N−2 chunk of data58may already be in the process of being written to the disk62by virtue of operation of the virtual ordered writes mechanism, discussed elsewhere herein.

Following the step562is a test step564which determines if the N−2 chunk of data58is empty. If not, control loops back to the test step564to continue to poll until the N−2 chunk of data58is empty. Otherwise, control transfers from the step564to a test step566which determines if the remote storage device26has received an acknowledge message from the local storage device24indicating that the local storage device24has transmitted all of the N−1 chunk of data54on the local storage device. If the acknowledge message has not been received, then control loops back to poll until the acknowledge message is received. Otherwise, control transfers from the step566to a step568where the remote storage device26begins to write the data from the N−1 chunk of data56to the disk62. At the end of the process illustrated by the flow chart560, the chunks of data56,58on the remote storage device26should be empty or should be in the process of being emptied.

Referring toFIG. 16, a flow chart600illustrates steps performed in connection with receiving a write from the host22(or from another entity) at the local storage device24during a transition from virtual ordered writes mode to synchronous or semi-synchronous RDF transfer mode. Processing begins a first step602which determines if the N−1 chunk of data54at the local storage device24is empty. If not, then control transfers from the test step602to a step604where the data being written is added to the N chunk of data52in a manner consistent with virtual ordered writes mechanism, discussed elsewhere herein. As discussed above, synchronous or semi-synchronous RDF transmission of data may not be begin until the N−1 chunk of data54is empty. Thus, the step604represents adding the data to the N chunk of data52when the system is not yet ready to begin synchronous or semi-synchronous RDF transmission. Following the step604, processing is complete.

If it is determined at the step602that the N−1 chunk of data54is empty, control transfers to a test step606which determines if the N chunk of data52is empty. If so, then control transfers from the step606to a step608where the data that is being written is transferred to the remote storage device26using synchronous or semi-synchronous RDF transfer mode. Following step608, processing is complete.

If it is determined at the step606that the N chunk of data52is not empty, then control transfers from the step606to a test step612which determines if the data being written is for the same slot as data stored in the N chunk of data52. If not, then control transfers from the test step612to the step608where, as described above, the data is transferred to the remote storage device26using synchronous or semi-synchronous RDF transmission. Thus, if a data write occurs and the data being written is not related to any data that is in the N chunk of data52, then the data may be written using synchronous or semi-synchronous RDF transmission.

If it is determined at the test step612that the data that is being written corresponds to data in a slot that is in the N chunk of data52, then control transfers from the step612to a step614where the slot in the N chunk of data52is locked to prevent other accesses thereto. Following the step614is a step616where the data in the N chunk of data is merged with the data being written, with the data being written taking precedence. Merging the data at the step616is analogous to merging the data at the step314in the flow chart300ofFIG. 9, discussed above. Following the step616is a step618where the data from the merged slot is transferred to the remote storage device26using synchronous or semi-synchronous RDF transfer mode. Following the step618is a step622where the slot is unlocked and then removed from the N chunk of data52(i.e., a pointer to the slot is removed from the N chunk of data52). Note that once the data has been transferred at the step618, the slot is no longer needed and thus may be removed from the N chunk of data at the step622. Following step622, processing is complete.

Referring toFIG. 17, a flow chart650illustrates steps performed at the remote storage device26in connection with receiving data transferred thereto by the local storage device24using the synchronous or semi-synchronous RDF transfer mode. Processing begins at a first step652where it is determined if the received synchronous or semi-synchronous data is for a cycle number corresponding to the inactive buffer of the remote storage device26. If not, then control transfers from the test step652to a step654where a synchronous or semi-synchronous receive is performed. Following the step654is a step656where an acknowledgement is provided from the remote storage device26to the local storage device24to acknowledge the received data. Following the step656, processing is complete.

If it is determined at the test step652that the received data corresponds to the inactive buffer of the remote storage device26, then control transfers from the step652to a test step658where it is determined if the inactive buffer is currently being restored at the remote storage device26(see the step568in the flow chart560ofFIG. 15, discussed above). If not, then control flows from the step658to a step662where the received data is added to the inactive buffer. Following the step662is the step656, discussed above.

If it is determined at the step658that the inactive buffer is being restored at the remote storage device26, then control flows from the step658to a test step664where it is determined if the inactive buffer contains a slot that is related to the received data (i.e., for the same slot as the received data). If so, then control transfers from the step664to a step666where the related slot in the inactive buffer is restored to the disk62of the remote storage device26. Following the step666or following the step664if the inactive buffer does not contain a related slot is a step668where the received data is restored to the disk62of the remote storage device26. Following the step668is the step656, discussed above.

Other techniques may be used for transitioning from virtual ordered writes mode to synchronous or semi-synchronous RDF transfer mode. These techniques may combine some or all of the specific features of the techniques described above.

Referring toFIG. 18, a flow chart700illustrates steps performed for an alternative embodiment in connection with the local storage device24switching from virtual ordered writes mode to synchronous or semi-synchronous RDF transfer mode. The local storage device24may switch in response to a command from the host22or from some other entity. In some embodiments, the local storage device24may monitor data transfers and use particular metrics to automatically determine that a switch is appropriate.

Processing for the flow chart700begins at a first step702where it is determined if the number of data entries in the N chunk of data52(the active chunk) of the local storage device24is greater than one half of the maximum number of entries. If so, then control transfers from the test step702to a step704where waiting for a predetermined period of time (e.g., one second) is performed. Following the step704, control transfers back to the step702, discussed above. The steps702,704, are performed because it may be desirable to not perform the transition to synchronous or semi-synchronous RDF transfer mode unless the active buffer (the N chunk of data52) has a significant amount of unused space so that data that is added to the buffer in the course of the transition, as discussed below, does not overflow the buffer. Of course, the amount of space used for the test at the step702could be different than the one half illustrated herein. In addition, in some it may be possible to eliminate the step702(and associated steps) altogether and perform the transition irrespective of the amount of available space remaining in the active buffer (the N chunk of data52). Note also that it is possible to simply abort the transition rather than waiting and trying again if it is determined at the step702that the active buffer is more than ½ full. In such a case, the fact that the abort occurred may be reported back to the calling software, which may then decide whether to attempt to transition again.

If it is determined at the test step702that the active buffer (the N chunk of data52) has sufficient space, then control transfers from the test step702to a step706where a SYNC_STATE variable (SS) is set to one. Use of the SYNC_STATE variable is described elsewhere herein. Following the step706is a test step708which determines if the SYNC_STATE variable is greater than or equal to three. As described elsewhere herein, other processes increment and set the SYNC_STATE variable to different values in response to the SYNC_STATE not being equal to zero. If it is determined at the test step708that these other processes have not incremented the SYNC_STATE variable to three or greater, then the test step708loops back to continue to poll to wait for the SYNC_STATE variable to be greater than or equal to three. Otherwise, control transfers from the test step708to a step712where the virtual ordered writes state (and associated processes) are deactivated.

Following the step712is a test step714which determines if the SYNC_STATE variable has been set back to zero (by other processes). If not, control loops back on the test step714to continue polling. Otherwise, control transfers from the step714to a step716where the virtual ordered writes mode (and associated processes) are activated. Following the step716, processing is complete.

In an alternative embodiment, it is possible to eliminate the steps714,716altogether, in which case the virtual ordered writes mode may be reactivated directly by another process. In such a case, part of the initialization performed by the virtual ordered writes code could be to set the SYNC_STATE variable to zero. Alternatively, the other process that reactivates the virtual ordered writes code, rather than the virtual ordered writes code itself, could set the SYNC_STATE variable to zero.

Referring toFIG. 19, a flow chart720is similar to the flow chart350ofFIG. 10. Steps of the flow chart720that are the same as steps for the flow chart350ofFIG. 10have the same reference number. However, note that the flow chart720shows a test step722that follows the step364. At the test step722, it is determined if the SYNC_STATE variable equals zero. The SYNC_STATE variable being zero indicates non-transitioning steady state operation in the virtual ordered writes mode. Thus, if it is determined at the test step722that the SYNC_STATE variable equals zero, then control transfers from the step722to the step366to continue processing as discussed above in connection with the flow chart350ofFIG. 10. Control flow from the step722to the step366represents non-transitioning steady state operation in the virtual ordered writes mode.

If it is determined at the test step722that the SYNC_STATE variable does not equal zero, then control transfers from the step722to a step724where the SYNC_STATE variable is incremented. Following the step724, control transfers to the step366to continue processing as discussed above. In some embodiments, it is useful to atomically update the sequence number (incremented at the step366) and the SYNC_STATE variable (incremented at the step724). This may be provided in a number of ways including, for example, causing the process illustrated by the flow chart720to be uninterruptible from a time before the step366to a time after the step724. In addition, it may be useful to retest (like the test at the step702) whether the active buffer less than some percentage full (e.g., half) before transitioning from SYNC_STATE=one to SYNC_STATE=two. In embodiments that perform this extra test, it is possible to abort the transition or perhaps even wait.

Writes to the local storage device24are handled in a special way during the transition. Referring toFIG. 20, a flow chart740illustrates handling writes to the local storage device24during the transition from normal virtual ordered writes mode to the synchronous or semi-synchronous RDF transfer mode. Processing begins at a first test step742where it is determined if the SYNC_STATE variable is zero. If so, then control transfers from the step742to a step744where a virtual ordered write is performed as discussed elsewhere herein (see, for example, the flow chart140ofFIG. 5). As discussed elsewhere herein, the SYNC_STATE variable being zero indicates being in non-transitioning virtual ordered writes mode. Following the step744, processing is complete.

If it is determined at the test step742that the SYNC_STATE variable does not equal zero, than control transfers from the test step742to a test step746where it is determined if the SYNC_STATE variable is greater than two. If so, then control transfers from the test step746to a step747where a synchronous or semi-synchronous RDF transfer record is constructed to transfer the data from the local storage device24to the remote storage device26. Following the step747is a step748where a cycle number is appended to the synchronous or semi-synchronous RDF transfer record constructed at the step747. Use of the appended cycle number is described elsewhere herein. Following the step748is a step752where a synchronous or semi-synchronous RDF transfer is performed. Following the step752, processing is complete. Note that it is possible for different processes and even different processors to perform different parts of the processing illustrated by the flow chart740so that, for example, the HA28may perform the steps747,748while one of the RA's30a–30cperforms the step752.

If it is determined at the test step746that the SYNC_STATE variable is not greater than two (i.e., that the SYNC_STATE variable equals one or two), then control transfers from the test step746to a test step754where it is determined if the data being written is for a slot that is already in the inactive buffer (the N−1 chunk of data54) of the local storage device24. If not, then control transfers from the test step754to the step747, discussed above.

If it is determined at the test step754that the data being written is for a slot already in the inactive buffer (the chunk of data54) of the local storage device24, then control transfers from the step754to a step756where the slot is locked. Following the step756is a step758where the data being written is merged, overwritten, or partially overwritten with the slot in the inactive buffer (the chunk of data54) of the local storage device24in a manner discussed elsewhere herein. Following the step758is step762where the slot is unlocked. Following the step762, processing is complete.

Referring toFIG. 21, a flow chart800illustrates steps performed in connection with the remote storage device26receiving synchronous or semi-synchronous RDF transferred data. Processing begins at a test step802, where it is determined if the received data corresponds to the N−1 chunk of data56at the remote storage device26. The test at the step802is made using the cycle number appended at the step748, discussed above. Thus, if a cycle switch occurs after the data is sent by the local storage device24but before the data is received and processed by the remote storage device26, use of the appended cycle number is designed to remove the possibility of improperly processing the data.

If it is determined at the test step802that the received data does not correspond to the N−1 chunk of data56at the remote storage device26, then control transfers to a test step804where it is determined if the received data corresponds to the N−2 chunk of data58of the remote storage device26. The step804also uses the appended cycle number. If it is determined at the test step804that the received data does not correspond to the N−2 chunk of data58at the remote storage device26, then control transfers to a step806where a synchronous or semi-synchronous receive is performed (i.e., steps performed in steady state J0/J1RDF mode).

If it is determined at the test step802that the received data corresponds to the N−1 chunk of data56at the remote storage device26, then control transfers to a step808where the received data is added to the N−1 chunk of data56in a manner consistent with the description elsewhere herein. Following the steps806,808is a step812where a synchronous or semi-synchronous acknowledgement is provided from the remote storage device26to the local storage device24. Following the step812, processing is complete.

If it is determined at the step804that the received data corresponds to the N−2 chunk of data58, then control transfers from the step804to a test step814where it is determined if the N−2 chunk of data58already contains corresponding data (i.e., data for the same slot). If so, control transfers from the step814to a step816where the corresponding slot in the N−2 chunk of data is restored. Following the step816, or following the step814if there is no corresponding data, is the step806, discussed above.

As for transitioning from synchronous or semi-synchronous RDF transfer mode to virtual ordered writes mode, it is simply a matter of initiating virtual ordered writes and beginning to accumulate data in the initial chunk at the local storage device24.

While the invention has been disclosed in connection with various embodiments, modifications thereon will be readily apparent to those skilled in the art. Accordingly, the spirit and scope of the invention is set forth in the following claims.