Data re-MRU to improve asynchronous data replication performance

A method for improving asynchronous data replication between a primary storage system and a secondary storage system is disclosed. In one embodiment, such a method includes monitoring, in a cache of the primary storage system, unmirrored data elements needing to be mirrored, but that have not yet been mirrored, from the primary storage system to the secondary storage system. The method maintains an LRU list designating an order in which data elements are demoted from the cache. The method determines whether a data element at an LRU end of the LRU list is an unmirrored data element. In the event the data element at the LRU end of the LRU list is an unmirrored data element, the method moves the data element to an MRU end of the LRU list. A corresponding system and computer program product are also disclosed.

BACKGROUND

Field of the Invention

This invention relates to systems and methods for improving transfer performance in asynchronous data replication environments.

Background of the Invention

In asynchronous data replication environments such as z/OS Global Mirror (also referred to as “XRC”) and Global Mirror, data is asynchronously mirrored from a primary storage system to a secondary storage system to maintain two consistent copies of the data. The primary and secondary storage systems may be located at different sites, perhaps hundreds or even thousands of miles away from one another. In the event an outage occurs at the primary storage system, host I/O may be redirected to the secondary storage system, thereby enabling continuous operations. When the outage is corrected or repaired at the primary storage system, host I/O may be redirected back to the primary storage system.

In asynchronous data replication environments such as XRC, updated data elements (e.g., tracks) are written to cache of the primary storage system. The updated data elements are recorded in an out-of-sync bitmap (i.e.,00S) to indicate that they need to be mirrored to the secondary storage system. Data elements that are written to the primary cache may be destaged to backend storage drives residing on the primary storage system, and eventually demoted. The destage and demotion processes are independent from the asynchronous mirroring process.

In certain cases, the primary storage system may destage and demote a data element before the asynchronous mirroring takes place. In such cases, the data element may need to be re-staged to the primary cache so it can then be mirrored to the secondary storage system. In other cases, the data element may be asynchronously mirrored to the secondary storage system before the data element is destaged and demoted from the primary cache. This scenario is preferred, since it only requires mirroring modified portions (e.g., sectors) of the updated data element to the secondary storage system, whereas a scenario that re-stages the data element to the primary cache not only requires moving the data element from the backend storage drives to the primary cache, but also requires mirroring the entire data element (e.g., track) to the secondary storage system. Thus, re-staging and mirroring an unmirrored data element may be significantly less efficient than mirroring the data element prior to its destage and/or demotion.

SUMMARY

The invention has been developed in response to the present state of the art and, in particular, in response to the problems and needs in the art that have not yet been fully solved by currently available systems and methods. Accordingly, the invention has been developed to improve asynchronous data replication between a primary storage system and a secondary storage system. The features and advantages of the invention will become more fully apparent from the following description and appended claims, or may be learned by practice of the invention as set forth hereinafter.

Consistent with the foregoing, a method for improving asynchronous data replication between a primary storage system and a secondary storage system is disclosed. In one embodiment, such a method includes monitoring, in a cache of the primary storage system, unmirrored data elements needing to be mirrored, but that have not yet been mirrored, from the primary storage system to the secondary storage system. The method maintains an LRU list designating an order in which data elements are demoted from the cache. The method determines whether a data element at an LRU end of the LRU list is an unmirrored data element. In the event the data element at the LRU end of the LRU list is an unmirrored data element, the method moves the data element to an MRU end of the LRU list.

A corresponding computer program product and system are also disclosed and claimed herein.

DETAILED DESCRIPTION

The present invention may be embodied as a system, method, and/or computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

The computer readable program instructions may execute entirely on a user's computer, partly on a user's computer, as a stand-alone software package, partly on a user's computer and partly on a remote computer, or entirely on a remote computer or server. In the latter scenario, a remote computer may be connected to a user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.

Referring toFIG. 1, one example of a network environment100is illustrated. The network environment100is presented to show one example of an environment where embodiments of the invention may operate. The network environment100is presented only by way of example and not limitation. Indeed, the systems and methods disclosed herein may be applicable to a wide variety of different network environments in addition to the network environment100shown.

As shown, the network environment100includes one or more computers102,106interconnected by a network104. The network104may include, for example, a local-area-network (LAN)104, a wide-area-network (WAN)104, the Internet104, an intranet104, or the like. In certain embodiments, the computers102,106may include both client computers102and server computers106(also referred to herein as “hosts”106or “host systems”106). In general, the client computers102initiate communication sessions, whereas the server computers106wait for and respond to requests from the client computers102. In certain embodiments, the computers102and/or servers106may connect to one or more internal or external direct-attached storage systems112(e.g., arrays of hard-disk drives, solid-state drives, tape drives, etc.). These computers102,106and direct-attached storage systems112may communicate using protocols such as ATA, SATA, SCSI, SAS, Fibre Channel, or the like.

The network environment100may, in certain embodiments, include a storage network108behind the servers106, such as a storage-area-network (SAN)108or a LAN108(e.g., when using network-attached storage). This network108may connect the servers106to one or more storage systems110, such as arrays110aof hard-disk drives or solid-state drives, tape libraries110b, individual hard-disk drives110cor solid-state drives110c, tape drives110d, CD-ROM libraries, or the like. To access a storage system110, a host system106may communicate over physical connections from one or more ports on the host106to one or more ports on the storage system110. A connection may be through a switch, fabric, direct connection, or the like. In certain embodiments, the servers106and storage systems110may communicate using a networking standard such as Fibre Channel (FC) or iSCSI.

Referring toFIG. 2, one embodiment of a storage system110acontaining an array of storage drives204(e.g., hard-disk drives and/or solid-state drives) is illustrated. As shown, the storage system110aincludes a storage controller200, one or more switches202, and one or more storage drives204such as hard disk drives and/or solid-state drives (such as flash-memory-based drives). The storage controller200may enable one or more hosts106(e.g., open system and/or mainframe servers106) to access data in the one or more storage drives204. The storage drives204may, in certain embodiments, be configured in RAID arrays of various RAID levels to provide desired levels of I/O performance and/or data redundancy. Logical volumes302(as shown inFIG. 3) may be carved from these RAID arrays.

In selected embodiments, the storage controller200includes one or more servers206. The storage controller200may also include host adapters208and device adapters210to connect the storage controller200to host devices106and storage drives204, respectively. During normal operation (when both servers206are operational), the servers206may manage I/O to different logical subsystems (LSSs) within the enterprise storage system110a. For example, in certain configurations, a first server206amay handle I/O to even LSSs, while a second server206bmay handle I/O to odd LSSs. These servers206a,206bmay provide redundancy to ensure that data is always available to connected hosts106. Thus, when one server206afails, the other server206bmay pick up the I/O load of the failed server206ato ensure that I/O is able to continue between the hosts106and the storage drives204. This process may be referred to as a “failover.”

In selected embodiments, each server206includes one or more processors212and memory214. The memory214may include volatile memory (e.g., RAM) as well as non-volatile memory (e.g., ROM, EPROM, EEPROM, flash memory, local disk drives, local solid state drives etc.). The volatile and non-volatile memory may, in certain embodiments, store software modules that run on the processor(s)212and are used to access data in the storage drives204. These software modules may manage all read and write requests to logical volumes302in the storage drives204.

In selected embodiments, the memory214includes a cache218, such as a DRAM cache218. Whenever a host106(e.g., an open system or mainframe server106) performs a read operation, the server206that performs the read may fetch data from the storages drives204and save it in its cache218in the event it is required again. If the data is requested again by a host106, the server206may fetch the data from the cache218instead of fetching it from the storage drives204, saving both time and resources. Similarly, when a host106performs a write, the server106that receives the write request may store the write in its cache218, and destage the write to the storage drives204at a later time. When a write is stored in a cache218, the write may also be stored in non-volatile storage (NVS)220of the opposite server206so that the write can be recovered by the opposite server206in the event the first server206fails.

One example of a storage system110ahaving an architecture similar to that illustrated inFIG. 2is the IBM DS8000® enterprise storage system. The DS8000® is a high-performance, high-capacity storage controller providing disk and solid-state storage that is designed to support continuous operations. Nevertheless, the systems and methods disclosed herein are not limited to the IBM DS8000® enterprise storage system, but may be implemented in any comparable or analogous storage system or group of storage systems, regardless of the manufacturer, product name, or components or component names associated with the system. Any storage system that could benefit from one or more embodiments of the invention is deemed to fall within the scope of the invention. Thus, the IBM DS8000® is presented only by way of example and is not intended to be limiting.

Referring toFIG. 3, in certain embodiments the host systems106and storage systems110adescribed inFIGS. 1 and 2may be used in a data replication environment, such as an asynchronous data replication environment300. As previously mentioned, in asynchronous data replication environments such as z/OS Global Mirror (also referred to hereinafter as “XRC”) and Global Mirror, data is mirrored from a primary storage system304ato a secondary storage system304bto maintain two consistent copies of the data. The primary and secondary storage systems304a,304bmay each be a storage system110asuch as that illustrated inFIG. 2. The primary and secondary storage systems304a,304bmay be located at different sites, perhaps hundreds or even thousands of miles away from one another. In the event an outage occurs at the primary site, host I/O may be redirected to the secondary storage system304b, thereby enabling continuous operations. When the outage is corrected or repaired at the primary site, host I/O may be redirected back to the primary storage system304a.

FIG. 3is a high-level block diagram showing an asynchronous data replication environment such as XRC. Using XRC, updated data elements (e.g., tracks) are written to cache218of the primary storage system304a. The updated data elements are recorded in an out-of-sync bitmap (i.e.,00S) to indicate that they need to be mirrored to the secondary storage system304b. Data elements that are written to the primary cache218may be destaged to backend storage drives204(i.e., volumes302a) residing on the primary storage system304a, and eventually demoted from the primary cache218. The destage and demotion processes are independent from the asynchronous mirroring process. In certain cases, the primary storage system304amay destage and demote a data element before the asynchronous mirroring takes place. In such cases, the data element may need to be re-staged to the primary cache218so it can be mirrored to the secondary storage system304b.

In other cases, the data element may be asynchronously mirrored to the secondary storage system304bbefore the data element is destaged and demoted from the primary cache218. This scenario is preferred, since it only requires mirroring modified portions (e.g., sectors) of the updated data element to the secondary storage system304b, whereas a scenario that re-stages the data element to the primary cache218not only requires moving the data element from the backend storage drives204to the primary cache218, but also requires mirroring the entire data element (e.g., track) to the secondary storage system304b. Thus, re-staging and mirroring an unmirrored data element may be significantly less efficient than mirroring the data element prior to its destage and/or demotion.

Referring toFIG. 4, in certain embodiments, functionality may be provided within the primary storage system304ato improve asynchronous data replication between a primary storage system304aand a secondary storage system304b. More specifically, the functionality may retain more unmirrored data elements in the primary cache218so that the unmirrored data elements do not have to be retrieved from backend storage drives204prior to being mirrored to the secondary storage system304b. For the purposes of this disclosure, “unmirrored data elements” are defined to include data elements that need to be mirrored, but that have not yet been mirrored, from the primary storage system304ato the secondary storage system304b. For example, data elements that belong to mirroring relationships and have been modified on the primary storage system304a, but have not yet been mirrored to the secondary storage system304b, may be considered “unmirrored data elements.”

As shown inFIG. 4, in certain embodiments, in order to improve asynchronous mirroring performance between a primary storage system304aand a secondary storage system304b, a regular LRU list400may be maintained for a cache218residing on the primary storage system304a. This regular LRU list400may designate an order in which data elements are demoted from the primary cache218. The primary cache218may contain unmirrored data elements as well as data elements that do not need to be mirrored from the primary storage system304ato the secondary storage system304b(e.g., unmodified data elements or data elements that do not participate in mirroring relationships with the secondary storage system304b).

Referring toFIG. 5, while continuing to refer generally toFIG. 4, as data elements are demoted from the regular LRU list400, systems and methods in accordance with the invention may determine whether the data elements at the LRU end of the regular LRU list400are unmirrored data elements. If so, the systems and methods may move the unmirrored data elements back to a most recently used (MRU) end of the regular LRU list400(hereinafter referred to as “re-MRUing”) instead of demoting them, thereby providing the unmirrored data elements additional time in the primary cache218before they are demoted. This, in turn, provides additional time to asynchronously mirror the unmirrored data elements from the primary cache218to the secondary storage system304bwithout needing to restage the unmirrored data elements from backend storage drives204.

In certain embodiments, unmirrored data elements may only be moved to the MRU end of the regular LRU list400a certain number of times before they are demoted from the cache218. For example, in one embodiment, unmirrored data elements may only be re-inserted at the MRU end of the regular LRU list400a single time, not counting times the unmirrored data elements are added to the MRU end as a result of other accesses (e.g., reads and writes) not related to mirroring. Unmirrored data elements may be flagged to indicate that they have been re-added to the MRU end so that they are not re-added again. If an unmirrored data element is encountered at the LRU end of the regular LRU list400that has already been previously re-MRUed, the unmirrored data element may be demoted like other data elements.

FIG. 6shows one embodiment of a method600for demoting a data element from the primary cache218using the techniques illustrated and described in association withFIGS. 4 and 5. As shown, the method600determines602whether a demotion is needed to clear space in the cache218. If so, the method600determines604whether a number of unmirrored data elements in the cache218is above a threshold.

In general, if unmirrored data elements occupy more than a designated amount (e.g., twenty five percent) of the cache218, unmirrored data elements may be demoted as opposed to re-MRUed if they are encountered at the LRU end of the regular LRU list400. On the other hand, if unmirrored data elements occupy less than the threshold, the unmirrored data elements may be re-MRUed if not previously re-MRUed. Thus, at step604, the method600may branch in one of two different directions depending on whether a number of unmirrored data elements in the cache218is above or below a designated threshold.

If, at step604, a number of unmirrored data elements in the cache218is above the designated threshold, the method600demotes612, from the cache218, the data element at the LRU end of the regular LRU list400, regardless of whether the data element is an unmirrored data element. If, on the other hand, a number of unmirrored data elements in the cache218is at or below the threshold, the method600determines606whether the data element at the LRU end of the regular LRU list400is an unmirrored data element. If not, the method600demotes612the data element.

If, at step606, the data element at the LRU end of the regular LRU list400is an unmirrored data element, the method600determines608whether the unmirrored data element has been previously re-MRUed. If so, the method600demotes612the unmirrored data element from the cache218. If the unmirrored data element has not been previously re-MRUed, the method600determines610whether a re-MRU threshold has been reached. For example, if a re-MRU threshold is five hundred and a number of unmirrored data elements that have been re-MRUed during a cache demotion process has exceeded the re-MRU threshold, an unmirrored data element may be demoted612from the cache218regardless of its unmirrored status. If the re-MRU threshold has not been reached, the method600re-MRUes the unmirrored data element by reinserting it at the MRU end of the regular LRU list400.

FIG. 7shows an alternative embodiment of an environment that may be used to improve asynchronous data replication between a primary storage system304aand a secondary storage system304b. As shown, instead of using just a regular LRU list400, the illustrated environment includes a regular LRU list400and a transfer-pending LRU list700that are used in association with a cache218. The regular LRU list400designates an order in which data elements are demoted from the cache218. The transfer-pending LRU list700, by contrast, is a list of unmirrored data elements that will be demoted from the cache218after they have been transferred to the secondary storage system304b.

Referring toFIG. 8, while continuing to refer generally toFIG. 7, when space needs to be cleared in the cache218, a data element may be analyzed at an LRU end of the regular LRU list400. If the data element is an unmirrored data element, the unmirrored data element may be moved from the regular LRU list400to the transfer-pending LRU list700where it may reside until its mirrored to the secondary storage system304b. On the other hand, if the data element at the LRU end of the regular LRU list400is not an unmirrored data element, the data element may be demoted to clear space in the cache218.

When unmirrored data elements in the transfer-pending LRU list700are transferred from the cache218to the secondary storage system304b, the unmirrored data elements may be removed from the transfer-pending LRU list700and demoted from the cache218. By contrast, if an unmirrored data element in the regular LRU list400is transferred to the secondary storage system304b, the data element may remain in its current position in the regular LRU list400and be demoted in due course. If a cache hit (other than a mirror transfer to the secondary storage system304b) occurs to an unmirrored data element in the transfer-pending LRU list700or the regular LRU list400, the data element may be removed from its current positions on the transfer-pending LRU list700or regular LRU list400and moved to the MRU end of the regular LRU list400.

FIG. 9shows one embodiment of a method900that may be executed when space needs to be cleared in the cache218of the environment illustrated inFIGS. 7 and 8. As shown inFIG. 9, the method900initially determines902whether a demotion is needed to clear space in the cache218. If so, the method900determines904whether a data element at the LRU end of the regular LRU list400is an unmirrored data element. If not, the method900demotes906, from the cache218, the data element at the LRU end of the regular LRU list400. If the data element is an unmirrored data element, the method900transfers908the data element to the MRU end of the transfer-pending LRU list700.

When clearing space in the cache218using the regular LRU list400, if too many unmirrored data elements are encountered at the LRU end of the regular LRU list400, unmirrored data elements may be demoted from the cache218regardless of their unmirrored status.FIG. 10shows one embodiment of a method1000that may be executed in such a situation. As shown inFIG. 10, if, at step1002, a specified number of unmirrored data elements are encountered at the LRU end of the regular LRU list400when demoting data elements from the cache218, the method1000demotes1004, from the cache218, a specified number of unmirrored data elements from the LRU end of the transfer-pending LRU list700. As an example, if five hundred unmirrored data elements are encountered at the LRU end of the regular LRU list400during a cache demotion process, the method1000demotes1004, from the cache218, five hundred unmirrored data elements from the LRU end of the transfer-pending LRU list700. These numbers are presented by way of example and not limitation.

In an alternative embodiment, step1002ofFIG. 10may determine whether a size of the transfer-pending LRU list700exceeds a threshold. If so, the method1000may demote1004, from the cache218, a specified number of unmirrored data elements from the LRU end of the transfer-pending LRU list700. Alternatively, if a size of the transfer-pending LRU list700exceeds a threshold, the method1000may attempt to transfer a designated number of data elements from the cache218to the secondary storage system304band then demote the data elements from the cache218. If transfer is not possible due to bandwidth or link issues, the method1000may simply demote1004a specified number of unmirrored data elements from the LRU end of the transfer-pending LRU list700, or demote data elements from the LRU end of the transfer-pending LRU list700until the size of the transfer-pending LRU list700falls below the threshold.

FIG. 11shows yet another embodiment of an environment that may be used to improve asynchronous data replication between a primary storage system304aand a secondary storage system304b. In the illustrated embodiment, the cache218is a heterogeneous cache218comprising a higher performance portion218aand a lower performance portion218b. In one embodiment, the higher performance portion218ais made up of DRAM memory and the lower performance portion218bis made up of flash memory such as storage class memory (SCM). Within the lower performance portion218b, an area1100may be reserved for unmirrored data elements that are waiting to be transferred to the secondary storage system304b. A regular LRU list400may be maintained for the higher performance portion218aand may indicate an order in which data elements are demoted from the higher performance portion218a. A transfer-pending LRU list700may be maintained for the reserved area1100and may indicate an order in which unmirrored data elements are demoted from the reserved area1100when space needs to be cleared therein.

Referring toFIG. 12, while continuing to refer generally toFIG. 11, when space needs to be cleared in the higher performance portion218a, a data element at the LRU end of the regular LRU list400is checked to determine if it is an unmirrored data element. If the data element is an unmirrored data element, the data element is transferred from the higher performance portion218ato the reserved area1100within the lower performance portion218b. The unmirrored data element may also be removed from a directory (hash table) associated with the higher performance portion218aand added to a directory associated with the reserved area1100. Furthermore, the unmirrored data element may be removed from the LRU end of the regular LRU list400and added to the MRU end of the transfer-pending LRU list700.

Referring toFIG. 13, when space needs to be cleared in the higher performance portion218a, a method1300may be used to demote data elements from the higher performance portion218a. As shown, the method1300initially determines1302whether a demotion is needed from the higher performance portion218a. If so, the method1300determines1304whether the data element at the LRU end of the regular LRU list400is an unmirrored data element. In not, the method1300demotes1308, from the higher performance portion218a, the data element that is at the LRU end of the regular LRU list400.

If, at step1304, the data element at the LRU end of the regular LRU list400is an unmirrored data element, the method1300determines1306whether space is available in the reserved area1100. If not, the method1300demotes1310, from the reserved area1100, the unmirrored data element that is at the LRU end of the transfer-pending LRU list700. This will clear space in the reserved area1100to receive a new unmirrored data element. The method1300then moves1312the unmirrored data element from the higher performance portion218ato the reserved area1100, moves1312the unmirrored data element from the LRU end of the regular LRU list400to the MRU end of the transfer-pending LRU list700, and moves the unmirrored data element from the directory associated with the higher performance portion218ato the directory associated with the reserved area1100. If, at step1306, space is already available in the reserved area1100, the method1300performs each of the steps1312without clearing space in the reserved area1100.

Referring toFIG. 14, when asynchronous data replication functionality mirrors an unmirrored data element to the secondary storage system304b, a method1400may be performed to locate and mirror the unmirrored data element. As shown, the method1400determines1402whether it is time to mirror an unmirrored data element to the secondary storage system304b. If so, the method1400determines1404whether the unmirrored data element is in the higher performance portion218a. If so, the method1400transfers1410the unmirrored data element from the higher performance portion218ato the secondary storage system304b.

If the unmirrored data element is not in the higher performance portion218a, the method1400determines1406whether the unmirrored data element is in the reserved area1100. If so, the method1400transfers1412the unmirrored data element from the reserved area1100to the secondary storage system304band removes the unmirrored data element from the reserved area1100(which may include removing the unmirrored data element from the transfer-pending LRU list700and the directory associated with the reserved area1100). If the unmirrored data element is not in the higher performance portion218aor the reserved area1100, the method1400stages1408the unmirrored data element from backend storage drives204to the higher performance portion218a, transfers1408the unmirrored data element from the higher performance portion218ato the secondary storage system304b, and then demotes1408the data element from the higher performance portion218a.

The various techniques illustrated inFIGS. 4 through 14are advantageous in that they retain, as much as possible, unmirrored data elements in a cache218until the unmirrored data elements can be mirrored from the primary storage system304ato the secondary storage system304b. This reduces overhead associated with staging unmirrored data elements from backend storage drives204to the cache218. This also increases efficiency in that mirroring directly from the cache218to the secondary storage system304bonly requires mirroring modified portions of the data elements (e.g., modified sectors of tracks) to the secondary storage system304b, whereas staging the data element back to the primary cache218also requires mirroring the entire data element (e.g., the entire track) to the secondary storage system304b. The disclosed techniques are also advantageous in that they do not starve other data elements from the cache218. That is, if unmirrored data elements occupy too much of the cache218, the disclosed techniques may demote unmirrored data elements from the cache218regardless of whether the unmirrored data elements have been mirrored to the secondary storage system304b. Thus, the disclosed techniques balance LRU order with improved retention of unmirrored data elements.