Managing failures in mirrored systems

Provided are a method, system and program from for managing failures in a mirrored system. A copy relationship between primary and secondary storage locations, wherein updates to the primary storage locations are copied to the secondary storage locations. A failure is detected preventing an application from writing updates to the primary storage locations. A failure message is received for the application in response to detecting the failure, wherein the failure message is not provided to the application. The copying of updates to the primary storage locations to the secondary storage locations is suspended. The failure message is provided to the application in response to suspending the copying.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a method, system, and program for managing failures in mirrored systems.

2. Description of the Related Art

Disaster recovery systems typically address two types of failures, a sudden catastrophic failure at a single point in time or data loss over a period of time. In the second type of gradual disaster, updates to volumes may be lost. To assist in recovery of data updates, a copy of data may be provided at a remote location. Such dual or shadow copies are typically made as the application system is writing new data to a primary storage device. Different copy technologies may be used for maintaining remote copies of data at a secondary site, such as International Business Machine Corporation's (“IBM”) Extended Remote Copy (XRC), Coupled XRC (CXRC), Global Copy, and Global Mirror Copy. These different copy technologies are described in the IBM publications “The IBM TotalStorage DS6000 Series: Copy Services in Open Environments”, IBM document no. SG24-6783-00 (September 2005) and “IBM TotalStorage Enterprise Storage Server: Implementing ESS Copy Services with IBM eServer zSeries”, IBM document no. SG24-5680-04 (July 2004).

In data mirroring systems, data is maintained in volume pairs. A volume pair is comprised of a volume in a primary storage device and a corresponding volume in a secondary storage device that includes an identical copy of the data maintained in the primary volume. Primary and secondary storage controllers may be used to control access to the primary and secondary storage devices. In certain backup system, a sysplex timer is used to provide a uniform time across systems so that updates written by different applications to different primary storage devices use consistent time-of-day (TOD) value as a time stamp. Application systems time stamp data sets when writing such data sets to volumes in the primary storage. The integrity of data updates is related to insuring that updates are done at the secondary volumes in the volume pair in the same order as they were done on the primary volume. The time stamp provided by the application program determines the logical sequence of data updates.

In many application programs, such as database systems, certain writes cannot occur unless a previous write occurred; otherwise the data integrity would be jeopardized. Such a data write whose integrity is dependent on the occurrence of previous data writes is known as a dependent write. Volumes in the primary and secondary storages are consistent when all writes have been transferred in their logical order, i.e., all dependent writes transferred first before the writes dependent thereon. A consistency group has a consistency time for all data writes in a consistency group having a time stamp equal or earlier than the consistency time stamp. A consistency group is a collection of updates to the primary volumes such that dependent writes are secured in a consistent manner. The consistency time is the latest time to which the system guarantees that updates to the secondary volumes are consistent. Consistency groups maintain data consistency across volumes and storage devices. Thus, when data is recovered from the secondary volumes, the recovered data will be consistent.

Consistency groups are formed within a session. All volume pairs assigned to a session will have their updates maintained in the same consistency group. Thus, the sessions are used to determine the volumes that will be grouped together in a consistency group. Consistency groups are formed within a journal device or volume. From the journal, updates gathered to from a consistency group are applied to the secondary volume. If the system fails while updates from the journal are being applied to a secondary volume, during recovery operations, the updates that did not complete writing to the secondary volume can be recovered from the journal and applied to the secondary volume.

Certain applications, such as database applications, may write user data to one set of primary volumes in a session and write exception information to another set of primary volumes in another or the same session. If a failure occurs such that the application cannot continue to write to the primary volumes including the user data, the application may still be able to write exception information on the failure to different primary volumes having the exception information and this failure exception information may be propagated to the secondary volumes mirroring the exception information. In such case; the secondary volumes have error free user data, however the exception information for the user data in the secondary volumes indicates that a failure occurred. During failure recovery operations, the administrator must perform extensive recovery operations at the secondary site to correct this data discrepancy in the mirrored copy because the secondary copy of the exception information indicates a failure or error that is not reflected in the mirrored user data.

For these reasons there is a need in the art for improved techniques for handling failures in a mirrored environment.

SUMMARY

Provided are a method, system and program from managing failures in a mirrored system. A copy relationship between primary and secondary storage locations, wherein updates to the primary storage locations are copied to the secondary storage locations. A failure is detected preventing an application from writing updates to the primary storage locations. A failure message is received for the application in response to detecting the failure, wherein the failure message is not provided to the application. The copying of updates to the primary storage locations to the secondary storage locations is suspended. The failure message is provided to the application in response to suspending the copying.

In a further embodiment, a data mover asynchronously copies updates to the primary storage locations to the secondary storage locations in the relationship.

In a further embodiment, the detected failure comprises a failure of at least one of a storage device in which the primary storage locations are configured, a control unit providing access to the storage device, or a component in a fabric providing communication between the application and the storage device.

In a further embodiment, the failure message is generated by a host operating system including the application or a control unit operating system managing access to the primary storage locations.

In a further embodiment, the primary and secondary storage locations comprise first primary and secondary storage locations. The application further writes exception information to second primary storage locations that is copied to second secondary storage locations. The detected failure comprises a failure related to an accessibility of the first primary storage locations, and wherein suspending the copying of updates prevents information on the detected failure written to the exception information in the second primary storage locations from being copied to the second secondary storage locations.

In a further embodiment, updates not copied from the primary storage locations to the secondary storage locations are journaled during the suspension. A data recovery operation is performed by copying the data at the secondary storage locations to primary storage locations and the journaled updates are applied to the primary storage locations in response to copying the data at the secondary storage locations to the primary storage locations.

In a further embodiment, the failure is detected with respect to a first set of primary storage locations. The first set of primary storage locations and a second set of primary storage locations are in one consistency group. The suspension of copying of updates applies to copying updates to the first and second sets of primary storage locations to a corresponding first and second sets of secondary storage locations.

In a further embodiment, the suspension of copying does not apply to copying updates to a third set of primary storage locations to a corresponding third set of secondary storage locations. The third set of primary storage locations are not in the consistency group including the first and second sets of primary storage locations.

In a further embodiment, the first and second set of primary and secondary storage locations comprise different sessions. A first data mover is used to copy updates to the first set of primary storage locations to the first set of secondary storage locations and a second data mover is used to copy updates to the second set of primary storage locations to the second set of secondary storage locations.

In a further embodiment, I/O requests to the primary storage locations are quiesced in response to detecting the failure. The I/O requests are allowed to proceed against the primary storage locations in response to suspending the copying of the updates. Updates to the primary storage locations are indicated in response to allowing the I/O requests to proceed while the copying of the updates is suspended.

In a further embodiment, information related to the failure message is written to the primary storage locations during the suspension but is not copied to the secondary storage locations. The secondary storage locations do not include the information related to the failure message written to the primary storage locations.

DETAILED DESCRIPTION

FIG. 1illustrates an embodiment of a network computing environment. A network2includes a plurality of primary control units4a. . .4n; primary storages6a. . .6n; data movers8a. . .8nmanaging the copying of updates to the primary storages6a. . .6nto the secondary control units10a. . .10nand corresponding secondary storages12a. . .12n; a host14writing updates to the primary storages6a. . .6n; a monitor system16monitoring failures in the availability of the primary storages6a. . .6nto the host14; a system timer18; and a master data set20. The components4a. . .4n,6a. . .6n,8a. . .8n,10a. . .10n,12a. . .12n,14,16,18, and20are connected to the network2and the network2enables communication among these components. The network2may include one or more switches to provide one or more paths of communication between the different network2elements.

System data mover (SDM) programs8a. . .8nread updates from the primary storages6a. . .6nand form consistency groups of updates from the primary storage6a. . .6nto write to the corresponding secondary storage12a. . .12n. At the primary control units4a. . .4n, updates may be written to a side file in a cache. The updates may then be transferred to journals22a. . .22nmaintained by the SDMs8a. . .8n. Within each of the journals22a. . .22n, the updates are arranged into consistency groups. The journals14a. . .14nmay store one or more consistency groups. A consistency group has a consistency time for all data writes in a consistency group having a time stamp equal or earlier than the consistency time stamp. A consistency group is a collection of updates to the primary volumes such that dependent writes are secured in a consistent manner. The consistency time is the latest time to which the system guarantees that updates to the secondary volumes are consistent. Consistency groups maintain data consistency across volumes and storage devices. Thus, when data is recovered from the secondary volumes, the recovered data will be consistent.

Consistency groups are formed within a session. A session may comprise the operations of a primary-secondary volume pairs managed by one or more SDMs8a. . .8nand the volume pairs managed by the mirror program4. All volume pairs assigned to a session will have their updates maintained in the same consistency group. Thus, the sessions are used to determine the volumes that will be grouped together in a consistency group. If the system fails while updates from the journal22a. . .22nare being applied to a secondary volume, during recovery operations, the updates that did not complete writing to the secondary volume can be recovered from the journal and applied to the secondary volume.

The SDMs8a. . .8nmay comprise programs implemented in a system. The SDMs8a. . .8nmay be implemented at the primary control units4a. . .4n, secondary control units10a. . .10nor an independent site and system.

The master data set20includes a list of the sessions being managed and for each managed session, the time of the most recent update to a volume in such session. The master data set20may reside on volumes in the storages6a. . .6n,12a. . .12n. The journal data sets for a primary/secondary control pair may reside on any device. InFIG. 1, each SDM8a. . .8nis part of one session whose information is indicated in the master data set20. Each session may be assigned one or more SDMs8a. . .8n.

The system timer18provides the time stamps for updates to insure that a common time is used across all SDMs8a. . .8nto provide a common time reference for application programs writing updates to volumes to insure that updates are not mirrored out-of-sequence. Once updates in the journal22a. . .22nare organized within a consistency group, then the updates within a consistency are applied to the secondary storages12a. . .12n. The creation of consistency groups guarantees that the system will shadow data to a remote site in real time with update sequence integrity for any type of data. Using consistency groups ensures that updates applied to the secondary storages12a. . .12nwithin a session are consistent as of the consistency time of the consistency group and consistent between sessions. If a failure occurs while updates are written from the journal22a. . .22nto secondary volumes in the secondary storage12a. . .12n, then during recovery, the updates that were interrupted during system failure can be recovered from the journal and reapplied to the secondary volume. In this way, data is insured consistent within and across sessions during recovery as of a point in time. The copy operations use the master data set20to maintain consistency across sessions, such as International Business Machine Corporation's (“IBM”) Extended Remote Copy (XRC), Coupled XRC (CXRC), Global Copy, Global Mirror Copy, and synchronous mirroring such as Peer-to-Peer Remote Copy (PPRC).

The network2may comprise a Storage Area Network (SAN), Local Area Network (LAN), Intranet, the Internet, Wide Area Network (WAN), peer-to-peer network, arbitrated loop network, etc. The storages6a. . .6n,12a. . .12nmay comprise an array of storage devices, such as a Just a Bunch of Disks (JBOD), Direct Access Storage Device (DASD), Redundant Array of Independent Disks (RAID) array, virtualization device, tape storage, flash memory, etc.

In certain embodiments, the host operating system26and the primary control unit operating systems28a. . .28ninclude code to communicate certain predefined error notifications to the monitor system16before the error notification is sent to the application24. In this way, the monitor system16handles information on failures detected by the operating systems26and28a. . .28nbefore the application24. The secondary control units10a. . .10nalso include operating systems30a. . .30n.

The monitor system16may be implemented within one of the control units or in a separate system, such as shown inFIG. 1.

The host4includes one or more applications24that communicate I/O requests to the primary control unit4a. . .4n.FIG. 2illustrates an embodiment known in the prior art where the host4includes a database application40. The database application writes user data, i.e., database records, to the primary storage6avia the primary control unit4ato update a database tablespace42storing the tables and records of the database. If the host4receives or detects a failure of the connection to the primary storage4aor is unable to complete a write and signals the database application40of such error, then the database application40writes information on the error to an exception table44in primary storage6bvia primary control unit4b.

FIG. 3illustrates an embodiment of operations performed by the SDMs8a. . .8ncopying data for different sessions using an asynchronous remote copy technology, e.g., XRC, CXRC, etc. In response to initiating remote copy operations (at block100), the SDMs8a. . .8nform (at block102) consistency groups having updates to primary storage locations. As discussed, the SDM8a. . .8nmay form the consistency groups by reading the updates from the primary storages6a. . .6nto the journal22a. . .22nwhere the consistency group is formed. The SDMs8a. . .8nindicate (at block104) in the master data set20a time of an update added to a consistency group in the journal22a. . .22n. The SDMs8a. . .8nmay periodically poll (at block106) the master data set20to determine a reference time (a minimum of the maximum of session update times). The SDMs8a. . .8ncopy (at block108) consistency groups in their journals26a. . .26nto the secondary storages22a. . .22nwhose timestamp is less than or equal to the determined reference time. In an alternative embodiment, the data movers may synchronously copy data.

FIG. 4illustrates an embodiment of operations performed by the host operating system26in response to detecting (at block120) a failure that is a recognized trigger event to be handled first by the monitor system16. In response to such detection, the host operating system26communicates (at block122) the failure to the monitor system16. The host operating system26may detect a failure in the network2connection, e.g., switches, cables, etc., between the host14and the primary control units4a. . .4n.

FIG. 5illustrates an embodiment of operations performed by the primary control unit operating system28a. . .28nin response to detecting (at block130) a failure that is a recognized trigger event to be handled first by the monitor system16. In response to such detection, the control unit operating system28a. . .28ncommunicates (at block132) the failure to the monitor system16. The primary control unit operating systems28a. . .28nmay detect a failure in the network2connection to the host14or a failure in the primary storage6a. . .6nresources, e.g., disk, array or tape failure, or a failure in the connection between the primary storages6a. . .6nand their respective primary control units4a. . .4n.

With the embodiments ofFIGS. 4 and 5, a detected failure is intercepted by the monitor system16so that the monitor system16handles the failure message before the application24is notified of the failure. For instance, with respect to the database application40, intercepting the failure and having it routed to the monitor system16prevents the database application40from writing information on the failure to the exception table44, which in turn prevents information on the error/exception from being propagated to the secondary storage12bmirroring the primary storage4bhaving the exception table44. In this way, the monitor system16may monitor numerous different types of failure notifications being generated in the network2.

FIG. 6illustrates an embodiment of operations performed by the monitor system16to handle a failure communicated by the host26or primary control unit28a. . .28noperating systems. Upon receiving (at block150) information of a failure at a primary site (e.g., primary control unit or storage) from the host26or control unit28. . .28noperating system, the monitor system16issues commands to quiesce (at block152) I/O requests to the primary storage6a. . .6nvolumes. This command to quiesce may be provided to the primary control units4a. . .4nand/or the hosts14. The monitor system16then suspends (at block154) the copying of updates to all primary storage locations to the secondary storage locations in a consistency group including the storage locations affected by the failure. The monitor system16may cause the suspension by sending commands to the SDMs8a. . .8nto suspend forming consistency groups including the primary storage6avolumes or data sets for which failure was detected, so that any updates occurring after the failure is detected are not propagated to the secondary storages12a. . .12n. However, already formed consistency groups in the journals22a. . .22nmay be copied over to the secondary storages12a. . .12n.

In certain embodiments, the suspension may apply to all data that is dependent upon the primary storage6a. . .6nvolumes subject to the failure condition or which the primary storage6a. . .6nvolumes subject to the failure depend. This dependent data may comprise all data that is in the consistency group including the primary volumes subject to the failure condition, i.e., that have themselves failed or cannot be accessed due to a failure in a component connecting the host14to the primary storages6a. . .6n. Further, the dependent data may comprise volumes in other sessions whose consistency is maintained through the master data set20. Thus, the suspension may be issued with respect to all volumes in all sessions whose consistency is maintained with the failed volumes through the master data set20. Mirroring operations may continue to be performed with respect to those volumes not dependent on or within the consistency group of the volumes subject to the failure condition while other volumes are subject to the suspension.

After suspending I/O to the volumes in the consistency group including the volumes subject to the failure condition, the monitor system16may issue commands (at block156) to the primary control units4a. . .4nand host14to allow I/O requests to proceed against the primary storage6a. . .6nlocations. At this point, the failure message may be provided (at block158) to the application24. However, since the mirroring of updates has been suspended, any error condition the application24writes to a primary storage6a. . .6n, e.g., the database application40writing to the exception table44(FIG. 2), is not propagated to the secondary storage12a. . .12n. At this point, the primary control units4a. . .4nmaintain bitmaps for each volume to keep track of updates to the suspended primary storage6a. . .6nlocations.

FIG. 7illustrates operations to copy the data in the secondary storage12a. . .12n, which does not reflect the exceptions, to the primary storage6a. . .6nafter the failure that initiated the failure message resulting in the operations ofFIG. 5is addressed. At block200, the cause of the failure is fixed, which may involve servicing components in the network2, host14, primary control unit4a. . .4n, primary storage6a. . .6n, etc. After the failure is addressed, the monitor system16or administrator may initiate (at block202) a recovery operation to copy all secondary volumes or data sets involved in the suspension to the corresponding the primary storage6a. . .6nlocations. This may involve copying data from secondary volumes in different secondary storage devices, managed by different secondary control units and/or in different sessions. After the data is recovered from the secondary sites, any journaled updates to the suspended primary storage6a. . .6nlocations are applied (at block204) to the updated primary storage locations (i.e., updates not included in a formed consistency group). These may comprise updates in the journals22a. . .22nnot included in a consistency group or updates maintained in a side file of the primary control units4a. . .4n. The suspension of mirroring and forming consistency groups for the suspended volumes is ended (at block206) and mirror operations may continue.

With the described embodiments, indication of a failure, such as an exception written to the exception table44in a primary storage6bis not propagated to the secondary storage6bwhere related user data in another secondary storage6a, e.g., the mirrored tablespace42, may not have any errors. If the exception was propagated to the exception table being mirrored in the corresponding secondary storage12b, then the administrator would have to perform error recovery operations at the secondary storage12bto clear the error before recovering the tablespace42(FIG. 2) in the primary storage6afrom the secondary storage12a. However, because the described embodiments prevent the exception from being propagated to the mirrored exception table44, no recovery operations need be performed at the secondary site to clear a propagated exception. In this way, the described embodiments reduce downtime and simplify the failure recovery process.

ADDITIONAL EMBODIMENT DETAILS

The described operations may be implemented as a method, apparatus or article of manufacture using standard programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof. The described operations may be implemented as code maintained in a “computer readable medium”, where a processor may read and execute the code from the computer readable medium. A computer readable medium may comprise media such as magnetic storage medium (e.g., hard disk drives, floppy disks, tape, etc.), optical storage (CD-ROMs, DVDs, optical disks, etc.), volatile and non-volatile memory devices (e.g., EEPROMs, ROMs, PROMs, RAMs, DRAMs, SRAMs, Flash Memory, firmware, programmable logic, etc.), etc. The code implementing the described operations may further be implemented in hardware logic (e.g., an integrated circuit chip, Programmable Gate Array (PGA), Application Specific Integrated Circuit (ASIC), etc.). Still further, the code implementing the described operations may be implemented in “transmission signals”, where transmission signals may propagate through space or through a transmission media, such as an optical fiber, copper wire, etc. The transmission signals in which the code or logic is encoded may further comprise a wireless signal, satellite transmission, radio waves, infrared signals, Bluetooth, etc. The transmission signals in which the code or logic is encoded is capable of being transmitted by a transmitting station and received by a receiving station, where the code or logic encoded in the transmission signal may be decoded and stored in hardware or a computer readable medium at the receiving and transmitting stations or devices. An “article of manufacture” comprises computer readable medium, hardware logic, and/or transmission signals in which code may be implemented. A device in which the code implementing the described embodiments of operations is encoded may comprise a computer readable medium or hardware logic. Of course, those skilled in the art will recognize that many modifications may be made to this configuration without departing from the scope of the present invention, and that the article of manufacture may comprise suitable information bearing medium known in the art.