Zero-data loss recovery for active-active sites configurations

A computing system includes a first storage unit at a first computing site. The first storage unit stores units of work data and data synchronously replicated from a first server cluster of a second computing site. The system further includes a second server cluster at the first computing site, the second server cluster is a proxy node of the first server cluster. The system further includes a second storage unit at the first computing site, the second storage unit stores the units of work data and data from the first storage unit asynchronously into the second storage unit. The system further includes a third server cluster at the first computing site, the third server cluster processes the units of work data asynchronously replicated into the second storage unit.

BACKGROUND

The present invention relates to continuous availability between sites that are geographically separated from each other, and more specifically, to a multi-site continuous availability computing environment with a recover point objective (RPO) of zero seconds in case of an outage of a site.

In the past, some computer availability and disaster recovery solutions were limited to a maximum distance between sites. Other past solutions required starting systems, applications, and supporting infrastructure on the backup site that could in some cases take several hours to restart. Some past solutions additionally required modifications to software applications, such as database servers, and hardware, such as routers and switches, in order to implement various disaster recovery and continuous availability functions, resulting in relatively high implementation cost. Some past solutions operated at a site level, rather than at a workload level.

These issues have been substantially addressed by continuous availability solutions between sites at unlimited distances. However, it remains very difficult to be able to achieve an RPO of zero seconds when the sites are separated by relatively long distances.

SUMMARY

According to one or more embodiments of the present invention, a computing system includes a first storage unit at a first computing site, the first storage unit to store units of work data and data synchronously replicated from a first server cluster of a second computing site. The computing system further includes a second server cluster at the first computing site, the second server cluster is a proxy node of the first server cluster. The computing system further includes a second storage unit at the first computing site, the second storage unit to store the units of work data and data from the first storage unit asynchronously into the second storage unit. The computing system further includes a third server cluster at the first computing site, the third server cluster to process the units of work data asynchronously replicated into the second storage unit.

According to one or more embodiments of the present invention, a method includes, by a first storage unit of a first computing site, synchronously replicating units of work data and data that result from processing the units of work data by a first server of a second computing site. The method further includes, storing, by a second server of the first computing site, the units of work data and the data asynchronously from the first storage unit to a second storage unit of the first computing site, the second server is a proxy node of the first server of the second computing site. The method further includes, by a third server of the first computing site, processing the units of work data asynchronously replicated from the first storage unit.

According to one or more embodiments of the present invention, a computer program product for providing continuous availability includes a computer readable storage medium having stored thereon program instructions that are executable by one or more processing devices to cause the one or more processing devices to provide continuous computing availability for a system including a first computing site and a second computing site. A method for providing the continuous computing availability that is performed by the processing devices includes, synchronously replicating units of work data and data that result from processing the units of work data by a first server of a second computing site. The method further includes, storing, by a second server of the first computing site, the units of work data and the data asynchronously from the first storage unit to a second storage unit of the first computing site, the second server is a proxy node of the first server of the second computing site. The method further includes, by a third server of the first computing site, processing the units of work data asynchronously replicated from the first storage unit.

DETAILED DESCRIPTION

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Program/utility34, having a set (at least one) of program modules36, may be stored in memory26by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof may include an implementation of a networking environment. Program modules36generally carry out the functions and/or methodologies of embodiments of the invention as described herein.

With regard to the resource provisioning and service level management functions of the management layer90, some embodiments of the present invention provide continuous availability of workloads, disaster recovery, workload distribution and replication of application data across a plurality of sites. In particular, the embodiments of the invention provide a recover point objective (RPO) of zero seconds for the sites that are separated by relatively long distances.

Some existing availability systems are limited geographically and/or by recovery time. When one or more workloads are spread across multiple servers in a single location, the servers for each workload may share a single data repository, and all data related to each of workloads may be stored in the same location. When the workloads are split among geographically separated sites, a single data repository for each workload is not always feasible.

In these instances, data from the one or more workloads may be stored in a data repository at a primary site, and the data may be synchronized, between the primary site and a copy of the data at the secondary site. The time that it takes to synchronize the databases is called latency. As sites are spread further apart geographically, latency may increase because of the time it takes to move the data over a network in order to synchronize it. For instance, one millisecond of latency is added per 100 fiber kilometers between the sites. Once latency increases beyond a relatively small amount of time, transferring data between data centers requires increasingly longer periods of time to achieve synchronization.

As a result, some existing availability systems provide acceptable workload performance only within a limited geographic area. In some cases, this limited geographic area may be approximately 10 to 20 fiber kilometers (i.e., 10 to 20 linear kilometers of a fiber optic network).

Disaster recovery systems are designed to switch between a primary data center and a backup data center in situations where the primary data center becomes unavailable, such as, for example, during a power outage, or other unplanned site level component failure, for example UPS, cooling system etc. The primary data center may also become unavailable because of unplanned site failure caused by events like fire, earthquake, water leaking etc. For example, during normal operation all transactions may be distributed to the primary data center and the data may be periodically replicated bit-by-bit to the secondary site or sites.

Workloads generally may be executed in parallel on at least two distinct computing systems. Typically, at least two instances of a workload may be executed virtually simultaneously on at least two geographically separated computing systems, for example, an active instance executing on a computing system at a primary site and a standby instance executing on another computing system at a secondary site. Such a configuration may sometimes be referred to in the art as an active-active workload. But the existing Active-Active solution has a severe limitation that if one site crashes abruptly, the other site will lose some during taking over the transactions from the crashed site because of the asynchronous software data replication that is typically used. Such data loss is not acceptable, especially in critical applications such as banking, military applications etc., that require to ensure 24*7 non-interrupting data transaction continuity to maintain data consistency across both (or more) sites.

The distance between sites may include, for example, distances greater than the area covered within a metro area network (MAN), that is, a network that may span distances measured in tens of kilometers, for example, up to about 20 fiber kilometers. Some customers require that a primary site and a secondary redirection site be separated by distances sufficient to ensure that a disaster affecting one site is not likely to affect the other. Although these distances vary based on regional and environmental conditions, primary and secondary sites sometimes are separated by distances that extend beyond a MAN.

In some embodiments, the customer acceptability window may be measured by the length of an RPO. An RPO, as known in the art, is the unit of time up to which the secondary site's data is current after the primary site becomes unavailable. That is, an RPO defines the maximum targeted time period in which data might be lost after the primary site becomes unavailable. For example, the customer acceptable window may require an RPO of zero seconds of data loss when an unplanned interruption occurs.

In some embodiments, a workload may consist of one or more computing applications or jobs, as well as associated middleware runtime environments, data source objects used by the applications, and the network addressability of the applications. In some embodiments, a workload may consist of one or more computing applications, jobs or threads that are relatively time-sensitive and preferably will not be suspended at all, not even for a brief moment. In some embodiments, a workload includes a database, or a file system, a set of applications or resources that use, access and/or manage the database and/or file system.

A unit of work data may include one or more computing transactions and/or processes substantially performed as a group to service one or more requests. A unit of work data may include, for example, data generated by or otherwise associated with a single computing transaction and/or process, or with multiple computing transactions and/or processes substantially performed as a group to service one or more requests. A data object may include, for example, any combination of related or associated data.

In an embodiment, the continuous availability system may include a workload distribution module that collects metrics at the software application, middleware, operating system, network, and hardware levels for each workload. The continuous availability system may use the collected metrics to provide continuous availability and workload redirection capabilities across multiple computing sites.

Some embodiments of the invention provide systems and methods for achieving zero-data-loss recovery in an active-active sites configuration with a recovery time objective (RTO) measured in seconds, or at most a few minutes, for transactions that require data updates and sub-second for read-only transactions that can tolerate temperate data staleness, following an outage of a site. An RTO, as known in the art, is the maximum amount of time needed to begin normal operations after the primary site experiences an outage. The embodiments of the invention switch transactions to a geographically remote site where a remote read-only standby sharing workload coupled with a synchronous disk replication of recover logs is used for fast restart and for preventing data loss (zero RPO). Asynchronous log capture replication of the workloads to another data sharing parallel system is used for uninterrupted service. Further, the one or more embodiments of the invention described herein use an additional proxy node that is introduced to decouple the software data replication with the primary (source) site. In addition, a unified site switch procedure is executed in the various outage scenarios for performing the switch from the primary site to the backup site without data loss.

Accordingly, the one or more embodiments described herein introduce hardware synchronous replication method to ensure both sites log data evenly, then make use of software replication tool to read the active and archive log replicated from the database management systems of source site, and replay the log in the database management systems of the target (backup) site. The one or more embodiments of the invention accordingly facilitate an Active-Active solution without data loss.

With reference now toFIG. 4, a continuous availability system400is depicted for implementing continuous availability for workloads across multiple sites that are geographically separated, using existing techniques. The system400may include a workload distribution module402executing computer instructions. The workload distribution module402may operate in any type of environment that is capable of executing a software application. For example, the workload distribution module402may include a high-speed, multiuser, multitasking computer processing device, such as a mainframe computer. In some embodiments, the workload distribution module402may be associated with an enterprise (e.g., a commercial business) that implements the continuous availability across multiple sites that are geographically separated.

The continuous availability depicted inFIG. 4may include one or more computing sites, such as, for example, site one404and site two406. Each of the sites404,406may include one or more systems executing one or more workloads. The workloads may include transaction processing applications, database applications, queue and queue management operations, and the like. Each of the sites404and406may include, for example, one or more network hardware devices and/or software for managing and distributing network traffic.

Site one404and site two406may be geographically distributed computing sites. For example, site one404may be located in one region, for example region A416, and site two406may be located in another region, for example, region B418, that is relatively geographically distant from region A416. The geographic distance between region A416and region B418may provide for a relatively high probability that computer processing sites in region A416will not suffer outages, or otherwise become unavailable, at the same time as computer processing sites in region B418. In particular, the geographic distance between region A416and region B418may provide for a relatively high probability that computer processing sites in region A416and sites in region B418will not suffer outages, or otherwise become unavailable, due to a common cause, such as a regional power outage or natural disaster.

The continuous availability system400depicted inFIG. 4additionally may include a software replication module408. The software replication module408, may asynchronously replicate data for workloads between site one404and site two406. In one or more examples, the replication may be performed using Peer to Peer Remote Copy (PPRC), which is a protocol to replicate a storage volume424of the site one404to another storage volume434in the remote site two406. Further, in one or more examples, at least portions of the storage volume424are backed up using GDPS/PPRC HyperSwap (Geographically Dispersed Parallel Sysplex) to another disk426to provide continuous disk availability. Further, batch files, application logs, control files and other such information from the storage volume424is partially PPRCed to storage volume434of site two406for batch recovery. The storage volumes424and434may also be referred to as databases, or database management systems (DBMS), as they store the data for units of work and data that result from processing the units of work data by the sysplex(s).

A sysplex-A428of the site one404also uses a high-volume, low-latency replication (e.g. Q Replication (QREP), SQL Replication etc.) to replicate transactions to a sysplex B438of the site two406. The data of multiple Data Sharing Group (DSG) from a relational database management system (e.g. DB2 etc.)426are replicated from site one404to the site two406which is asynchronous software replication based on recover log read, publish and transactions replay using messaging or any other technique. The site two406also has a relational database management system436in which the replication is performed.

Here a ‘sysplex’ refers to a cluster of computing servers that are acting together as a single system image and facilitating multiple databases for direct reads and writes to shared data. It should be noted that although the embodiments of the present invention described herein use a sysplex, in other examples, other types of computing system at the sites404and406can be used. Alternatively, or in addition, the sysplex can include a group of servers, such as a server farm, operating on one or more workloads using local load balancing, or other methods of load distributing as is known in the art. In yet another embodiment, site one (and/or site two506)404may include multiple systems, each of which may execute one or more workloads. In various embodiments, site one404(and/or site two406) may include a combination of servers and server farms each operating on one or more workloads.

The workload distribution module402and the sites404and406may be communicatively coupled via one or more networks (not shown). The networks may be implemented using any type or combination of known networking device, including, but not limited to, a wide area network (WAN), a local area network (LAN), a global network (e.g., Internet), a virtual private network (VPN), an intranet and a telephone network. The networks may be implemented using a wireless network, a wired network, or any kind of physical network implementation known in the art.

The sites, such as site one404and site two406may be coupled to the workload distribution module402through multiple networks (e.g., intranet and Internet) such that not all of the sites are coupled to the workload distribution module402through the same network. The workload distribution module402may be implemented using one or more servers, for example, operating in response to a computer program stored in a storage medium accessible by the server.

In the continuous availability system400, units of work414initiated by users of the various systems or clients executing at the one or more sites may be distributed to one or more of the sites404and406through the workload distribution module402. The units of work414may be transmitted from systems outside of the sites404and406and may be processed as workloads within one or more of the sites.

It will be readily understood by a person of ordinary skill in the art that the execution of continuous availability across geographically dispersed sites described inFIG. 4may be implemented as modules in hardware, software executing on general-purpose hardware, or a combination thereof. Although only two sites are depicted inFIG. 4, it will be further understood that, in an embodiment, any number of sites may be implemented, and that any geographic distance may separate the sites. Furthermore, although the workload distribution module402is depicted as existing outside of the sites, it will be readily understood by a person of ordinary skill in the art that, in an embodiment, the workload distribution module402may be directly located at one or more of the sites.

The illustration ofFIG. 4is a simplified representation of the various components of the continuous availability system400for purposes of clarity. It will be understood by those of ordinary skill in the art, that additional or fewer components may be used in alternate embodiments. In additional embodiments, the layout and configuration of the components may differ from those ofFIG. 4without affecting the functionality of the continuous availability system400. In additional embodiments, the various components may be located in separate modules. In further embodiments, the functionality of various components may be incorporated into a single hardware or software module.

It is to be noted that, if one of the sites of the continuous availability system400, for example, site one404suffers outages, or otherwise becomes unavailable before a transaction at site one404is completed, this transaction may be lost before the unit of work data for the transaction get asynchronously replicated to site two406. This is because there is a replication delay at the time of the outage of site one404, generally on the order of sub-seconds, but possibly more if and when the outage or other system conditions affect replication delays. Therefore, the RPO for the continuous availability system400is non-zero, and probably as much time as the replication delay. Despite the non-zero RPO, such an asynchronous replication technique is employed by the continuous availability system400of some embodiments because the asynchronous replication technique allows unlimited distance between sites and avoids impacting transaction execution at a primary site for the transaction.

FIG. 5illustrates a continuous availability system500in accordance with some embodiments of the invention. The system500, unlike the system400described above, provides an active-active architecture without suffering any data loss (i.e., RPO of zero) when a primary site for a unit of work becomes unavailable. In some embodiments, the continuously availability system500includes all components of the system400ofFIG. 4. In addition, the continuous availability system500includes a proxy sysplex-C538at site two406and a proxy storage unit524at site two406.

The “proxy mode” capability allows the sysplex-C538to read log records from synchronously mirrored secondary volumes that can be several kilometers or tens of kilometers away from the source production group, for example sysplex-A428. In proxy mode the sysplex-C538can read a catalog of the source sysplex-A428, thus facilitating the proxy sysplex-C538to get access to the dictionaries by reading the source's compression dictionary data set (CDDS) via the mirrored secondary volumes. By moving the replication capture component from the source sysplex-A428to a new “proxy” sysplex (sysplex-C538), which is failure-isolated from the source sysplex-A428, clients of the system500can realize a zero data loss if the source site-A404experiences a failure and workloads are switched to the target site-B406.

The logs of multiple data sharing groups are synchronously replicated into site two406by long leg synchronous hardware replication from site one404to site two406, for example using Multi-Target Metro Mirror (MTMM) or any other such technique for hardware replication. As used herein, ‘RS’ is a disk location of replication storage of MTMM, with RS1being a primary disk (at site one404) of the MTMM replication; RS2is a local secondary disk of MTMM replication (at site one404itself), which is a full synchronous target copy of RS1; and RS3is a long leg replication storage of the MTMM replication (at site two406), and is a second synchronous target copy of RS1. Further, as used herein, ‘RL’ is a replication leg used by the MTMM replication, one replication leg being responsible for each synchronous replication of RS1into RS2and RS3. Here the relationship between RS1and RS2is called as RL1and the relationship between RS1and RS3is called as RL2or an MTMM long leg because RS3is the remote secondary replication storage. It should be noted that in other examples, different nomenclature may be used than the description here without changing the scope of the embodiments of the technical solutions described herein.

In one or more examples, the volumes of the MTMM long leg (RS3) are varied online and read only from the proxy sysplex-C538(and are not directly readable by sysplex-B438). Accordingly, a copy of database log of sysplex-A (RS1) from the database424is synchronously replicated (RL2) in the database524(RS3) of the sysplex-C538. In one or more examples, the proxy sysplex-C538creates a replica of the full volume database424of the source sysplex-A428at the site two406.

Further, the proxy sysplex-C538asynchronously replicates data sharing groups, such as using QREP, to capture the source DSGs log from the copy in the database524at site two406(RS3) into the database436of the sysplex-B438at the site two406. Such asynchronous replication facilitates the transactions from sysplex-A428to be replayed in the DSGs of the sysplex-B438. The asynchronous replication of the database524(RS3) to the database436of sysplex-B438can be performed by a software replication module508of the sysplex-C538.

Accordingly, data log captured (RS3) at the proxy sysplex-C538in the database524does not rely on the status of the source DSGs (RS1) and even if the source DSGs (RS1) are down (abnormally, planned etc.), the proxy data capture (RS3) at the proxy sysplex-C538works independently to (asynchronously) replicate the source data log of sysplex-A428to the sysplex-B438. Accordingly, the site switch can be completed with Recovery Point Objective 0 even if the source site one404is not operative because of planned/unplanned reasons.

In one or more examples, a copy of the source data log (RS1) is also synchronously stored in a secondary copy (RS2). The secondary copy is synchronously replicated in the database524(RS3) of the proxy sysplex-C538using MTMM (RL3). In one or more examples, such synchronous replication is performed redundantly and concurrently with the synchronous replication from the primary copy of the data log (RS1). Alternatively, the replication of the secondary copy (RS2) is initiated if the replication of the primary copy fails, for example due to failure in the synchronous replication communication (RL1). Such a secondary copy replication provides an additional failsafe to ensure an RPO of 0 in case of any failure at site one406.

FIG. 6depicts a flowchart of an example method for creating an active-active site failover system to enable zero data loss solutions according to one or more embodiments of the present invention. The method includes creating the sysplex C538the site two406as a proxy of the sysplex A428, which is at site one404(605). Creating the proxy sysplex C538includes providing the one or more computer servers of the sysplex C538access to the catalog of the database volume424of the sysplex A428. The catalog access enables the sysplex C538to access transaction log of the sysplex A428.

The method further includes synchronously replicating (mirroring) the database transaction log of the sysplex A428to the database524that is allocated to the proxy sysplex C538(610). The database replication capture component from the source production group (sysplex A428) is accordingly moved to the new “proxy” group that is failure-isolated from the source sysplex A428. This facilitates realizing a zero data loss solution when failing over from the source database group to the target group (sysplex B438).

In one or more examples, the sysplex C538and the sysplex B438may execute on the same set of computer servers at site two406, with sysplex C538configured on one or more logical partitions that are separate from one or more logical partitions assigned to execute the sysplex B438.

In one or more examples, the sysplex-C538executes a program, such as QCAP (Q capture) that captures transactions or row-level changes from source tables that are part of a subscription or publication, and then sends this transactional data as messages over queues. For example, the sysplex-C538includes a data sharing group (DSG-C). The capture program of sysplex-C538is attached to (subscribed) data storage424of the site one404. Accordingly, the sysplex-C538facilitates capturing logs using synchronous replication, such as MTMM long leg, from a copy (RS3) of the active log, archive log, and BSDS from DSG A of the sysplex-A428of site one404. The volumes of RS3are varied online and read in sysplex-C538because of the ‘proxy mode’ setting. Further, DSG-A compression dictionary and member status are replicated into RS3. The synchronous replication, (hardware replication) replicates the contents of the storage unit424simultaneously with or immediately after each write function to the storage unit524. In some embodiments, the hardware replication performs disk replication, copying or mirroring a complete image of the storage unit524. In some embodiments, the hardware replication replicates only disk I/O (input/output), and not memory I/O.

Further, the method includes asynchronous replication of the transaction log from the proxy sysplex C538to the sysplex B438, at620. The asynchronous replication may be performed using known techniques that were used to asynchronously replicate the transaction log from the sysplex A428to the sysplex B438across the two sites404and406. Because of the setup described herein (FIG. 5), the asynchronous replication is performed only within site two406, instead of across the sites. If the source DSG-A of the sysplex-A428develops a problem that causes a planned/unplanned outage, or downtime, or an abnormal abend, the capture program in the proxy sysplex-C538continues to operate and provide the logs of the DSG-A for transactions that have completed so far, and site switch to the sysplex B438completes successfully and gracefully (625,630).

FIG. 7depicts a method for performing a site switch according to one or more embodiments of the present invention. The site switch procedures are unified and completed within 1 min as thereby having a Recovery Time Objective (RTO) of less than 1 min. The method includes initiating a site switch from site-A404to site-B406(705). The switch includes switching the workloads being performed by the sysplex A428to the sysplex B438. The method for the site switch further includes quiescing the workloads queued at the sysplex A428(710). Further, the site A404is deactivated (720). The deactivation ensures that the workload distributor402does not assign further workloads to the sysplex A428.

Further, any database threads at sysplex A428are purged (725), to ensure that the data log in the database424is not accessed via site one404. Further yet, the site one404is ‘soft fenced’ to prevent unintended access to a device, set of devices in a logical subsystem (730). The soft fence can be configured using an operating system of the sysplex A428or any other known techniques.

The site switch method further includes stopping execution of the capture program in the proxy sysplex C538(735). The capture program facilitates the synchronous replication of the transaction log from the sysplex A428to the sysplex C538. Further, the sysplex B438is unfenced (740). Further yet, the sysplex B438is activated to enable the workload distributor402to assign workloads to the sysplex B438for execution (745). In one or more examples, a client is notified of the successful site switch, at750. The notification can be one or more of various types of electronic notifications, such as an email, text message, instant messenger, or any other type of audio-visual notification.

Referring toFIG. 6again, once the switch to sysplex B438is performed, the workload distributor402performs the workloads using the sysplex B438(635) until sysplex A428is operational again (640). Once the sysplex A428is operational, a switch is performed from sysplex B438to the sysplex A428(645). Once the switch to the sysplex A428is successfully completed, workloads are again performed using the sysplex-A428(650).

The method an active-active system500that facilitates switching gracefully from one site to another with RPO=0, RTO less than 1 minute, and with zero data loss. Table 1 provides a comparison of example scenarios of site switching of the active-active system500(without data loss) and active-active system that is presently used.

The one or more embodiments of the present invention facilitate an active-active system with site switch without data loss by decoupling data replication with a source site (site one404in examples herein). The one or more embodiments of the present invention accordingly ensure no data loss in case of planned or unplanned failure scenario at the source site. The one or more embodiments of the present invention accordingly are rooted in computer systems, such as server computers, and provide an improvement to the active-active sites that are used on various data and transaction intensive industry, such as that rely on Internet transactions, such as e-commerce, social media, banking, online gaming, and various other such industrial applications. The one or more embodiments of the invention described herein provide technical solutions to decouple site switch ability with transaction workload availability in Active/Active solution which ensures a graceful site switch successfully even if the source site major workload components run into a failure.

As described herein, a server cluster (sysplex-C538) is added as a proxy node of the source cluster (sysplex-A428) which contains database management systems (524) which are used by data replication. The data replication, in proxy node, are associated with the proxy database management systems (524) which read the updated logs from source database management systems (424) and replay the transactions in a target database management systems (436). The updated logs of source database management systems (424) are synchronously replicated into the proxy nodes database management system (524), such as by hardware replication, which ensures that the transaction logs are synchronous in both proxy node (538) and the source node (428).

If one or more component at the source site (404) such as DBMS instances (424) develop a failure, or an outage, the hardware synchronous data replication ensures that the proxy site logs (524) are the same as the data logs of the source node (424). Further, the asynchronous data replication in the proxy node (538) reads the duplicated logs from the proxy database management system (524) into the target database management system (436) without any impact from the failure/outage of the source DBMS instances (424). Accordingly, if a site switch is triggered, it can be completed gracefully.

The one or more embodiments of the present invention described herein accordingly facilitate a synchronous unit of work (UOW) data between database management systems in 2 or more sites without dependence on the primary site by decoupling a site switch ability with transaction workload availability in Active/Active solution which ensures the graceful site switch successfully even if the source site major workload components run into any issue.