Managing cluster split-brain in datacenter service site failover

A central controlling service for datacenter activation/deactivation control in a cluster deployment to assist in preventing a split-brain scenario. The central controlling service provides a central point of control in the datacenter for application servers to periodically query as to whether to go offline, online, or normal. Redundancy of the central service facilitates detection of datacenter failure by the redundant services interacting to resolve the state of control information. This control information is then used to answer the server queries. On startup from a datacenter failure, a single instance of the central service queries other redundant instance(s) to determine if the single instance is starting up from a datacenter-wide failure or from operations other than total datacenter failure. If the failure is datacenter-wide, a central service protocol assists in resolving to the single service keeping the associated datacenter servers offline; otherwise, the server queries are answered to go online.

BACKGROUND

Historically, computing clusters that span physical locations are challenged to differentiate a network failure between the two datacenters from an actual service failure in the active datacenter. One conventional solution to this problem is the addition of a third datacenter. The third datacenter is effectively a “witness” and can vote as to which of the two datacenters should have services up. However, while this provides a suitable solution, this also increases the networking and facility costs. Moreover, a weakness of this solution is that the WAN (wide-area network) networking failures can cause service outages in the active datacenter. It can be argued that customers should not experience a concurrent outage of both WAN networks, if deployed on independent hardware. An alternative solution is to create a second network connection between the two locations that is failure independent of the first. This also adds complexity to the deployment and it becomes difficult to determine what represents a failure-independent connection.

In a two-datacenter configuration customers are obligated to inject operational procedures into the solution. A solution can be created by manually activating the messaging solution in the passive datacenter (initially, datacenter2). However, this still does not address the behavior of the messaging solution in the active datacenter (initially, datacenter1). For example, a power failure in the active datacenter (datacenter1) can trigger the activation of the passive datacenter (datacenter2) messaging deployment; making it now the active datacenter. If the datacenter1has power restored without a connection to the datacenter2(or manual intervention) then the datacenter1will automatically return to service, thereby creating a “split-brain” condition. This is “split brain” because both datacenter1and datacenter2messaging solutions are in service.

A second aspect of the problem is managing site resilience for a large scale service deployment. In the large-scale deployment case the number of systems with which an operation team must interact limits the timeliness of the recovery. In a service environment maximizing service uptime is essential. Thus, the resource intensive aspect of the large-scale deployment increases downtime for the service, and thus, further degrades the customer experience.

SUMMARY

Disclosed is a mechanism that addresses the complexity of a solution and the three datacenter requirements of site resilience in a messaging deployment. The mechanism includes a datacenter activation coordinator (DAC) that when added to a cluster deployment assists in controlling activation of an associated datacenter and prevents inappropriate activations. The DAC provides a central point of control in the datacenter by making servers, (e.g., servers operating in an active/passive configuration) periodically query the DAC. In the passive datacenter (datacenter2, initially), the DAC coordinates messaging service activation in parallel across the respective servers. In the active datacenter (datacenter1, initially) the DAC can prevent the inappropriate activation of the original services, thereby preventing the split-brain situation.

The DAC serves as a central controlling entity for datacenter activation control by using a “mommy may I model” to control the datacenter servers that comprise the messaging solution, for example, particularly those datacenter servers that operate in an active/passive resource configuration. Additionally, activation of the passive datacenter is performed by modifying the data on the centralized (per datacenter) DAC. Moreover, the redundancy of the central controlling entity (the DAC) can be leveraged to automatically detect entire datacenter failures and therefore prevent the split-brain scenario. This is done by having the “allowed online” state to be reset when all DAC servers go offline at the same time. Subsequent “mommy may I” queries from datacenter servers receive a “stay offline” response until explicitly changed or full connectivity between the two datacenters is restored. Split brain is prevented by having the central entity block activation—when necessary—of the messaging resources that operate in an active/passive clustered model.

The solution activates the passive datacenter (datacenter2, initially) where a set of redundant servers exist to service requests when required; thus making it the active datacenter. By default the “mommy may I” answer is set to “do not activate”, until the passive datacenter is instructed to activate by an administrator. When a subsequent server query yields an answer to “activate”, the appropriate action (e.g., startup) is initiated, and the results are posted back to the DAC. By using a polling architecture where the datacenter server periodically querying the DACs, the solution can scale up. Changing the state of the servers involves only changing the state of the control tables for the DAC. Queries can be run against the data to determine what recovery action to take for the reported failures. The nature of the solution in the original active datacenter (datacenter1) is similar. The DAC is regularly queried to determine whether services and servers are to be up and providing service.

One characteristic of the set of redundant DACs is the startup behavior. Since a single DAC is part of a redundant server set at least one DAC server is running at all times. As part of DAC server startup, the DAC server queries the other DAC servers. If the other servers are all starting then the DAC server assumes a complete power failure and blocks all messaging service activation to the datacenter application servers, thereby preventing a split-brain scenario. This provides the protection required for the two datacenter configuration.

To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings. These aspects are indicative of the various ways in which the principles disclosed herein can be practiced, all aspects and equivalents of which are intended to be within the scope of the claimed subject matter. Other advantages and novel features will become apparent from the following detailed description when considered in conjunction with the drawings.

DETAILED DESCRIPTION

The disclosed architecture is central controlling service for datacenter activation/deactivation control in a cluster deployment to assist in preventing a split-brain scenario. The central controlling service provides a central point of control in the datacenter for application servers to periodically query as to whether to go offline or online. Redundancy of the central service facilitates detection of datacenter failure by the redundant services interacting to resolve the state of control information. This control information is then used to answer the server queries. On startup from a datacenter failure, a single instance of the central service queries other redundant instance(s) to determine if the single instance is starting up from a datacenter-wide failure or from operations other than total datacenter failure. If the failure is datacenter-wide, a central service protocol assists in resolving to the single service keeping the associated datacenter servers offline; otherwise, the server queries are answered to go online.

FIG. 1illustrates a computer-implemented system100for managing a datacenter. The system100includes an active request component102for receiving requests from an active query component104of active servers106of an active datacenter108for server control information110. The system100also includes an active response component112for sending answers to the active servers106(via the active query component104) to control the active servers106to prevent a split-brain scenario between the active datacenter108and a passive datacenter114.

The server control information110includes state indicating whether service is online or offline, and applies to each of the active servers106or a subset of the active servers106. The active servers106of the active datacenter108can be mailbox server applications, for example, or application servers, in general.

The active request component102, the active response component112, and the server control information110can be embodied as part of a central active coordination component116such that the active coordination component116can store the server control information110and manage the active servers106when a datacenter-wide failure occurs (e.g., power outage).

Similarly, the passive datacenter114includes a passive coordination component118that stores its server control information120for controlling server state of passive servers122.

The passive servers122include a passive query component124for sending requests to a passive request component126. The passive coordination component118can include the passive request component126and a passive response component128for sending answers to the passive servers122(via the passive query component124) to control the passive servers122to prevent a split-brain scenario between the active datacenter108and the passive datacenter114.

It is to be appreciated that there is a role reversal scenario between the active datacenter108and passive datacenter114based on the failure. For example, when the active datacenter108is operating properly, the passive datacenter114is in a “standby” mode. However, when the active datacenter108fails, the active datacenter108goes into the passive datacenter role and the passive datacenter114assumes the active datacenter role. When the failure of the offline datacenter (datacenter108) is corrected, an administrator can then choose to switch roles back by making the offline datacenter (datacenter1) the active datacenter, and taking the now active datacenter (datacenter2) to the passive (or offline) role.

In one specific implementation, the servers (106and122) of the datacenters provide mailbox application services. Some or all of the servers can operate in an active/passive configuration. The activation is coordinated to prevent concurrent activation of the servers (passive) in a same mode in two or more datacenters such as a split-brain scenario. Each datacenter (108or114) is configured with redundancy in the respective coordination component for monitoring state of the servers in the datacenter and changing state of the servers in the datacenter based on directives from the administrator or deduced desired state. The returning active datacenter108(after a fault such as a power failure) deduces its desired state to be offline via the active coordination component116. Each server of the active datacenter108queries the coordination component116for activation information that defines if each server (or groups of servers) of the datacenter108will be activated or deactivated. The activation information is stored in the coordination component as the server control information110.

In an alternative embodiment, a coordination component (e.g., active coordination component116) comprises redundant computing systems (as activation controllers) deployed in order to provide the desired availability of the datacenter server applications (e.g., mailbox server applications) at a more granular level, such as according to subsets of the servers. For example, the datacenter (e.g., active datacenter108) servers might be controlled on an individual or group basis, rather than a datacenter basis and their associated coordination component (e.g., active coordination component116) can be configured to manage datacenter services by servers, or groups of servers and/or services.

Multiple redundant computing systems (operating as redundant activation controllers) can be provided such that one set of redundant computing systems (activation controllers) communicates with a first subset of the servers and/or services of the datacenter, a second set of redundant computing systems (activation controllers) communicates with a second subset of the servers and/or services of the datacenter, and so on. Thus, if the first set subset of servers and/or services fails in the active datacenter, the coordination component facilitates activation of a corresponding passive set of servers and/or services, which can be accomplished manually by an administrator.

More specifically, the server applications106of the active datacenter108periodically poll the associated active coordination component116for control information. The active coordination component116continues to provide the appropriate answer based on state of the active datacenter108. For example, the active coordination component116can return an answer based on an administrator previously either inputting the answer to the active coordination component116or the active coordination component116computes the answer (or response). In the alternate embodiment the answer might be different per server.

FIG. 2illustrates a system200that employs datacenter activation coordinators (DACs) for the coordination of active and passive messaging datacenters. The system200includes an active coordination component202for controlling message services204of an active datacenter206, and a passive coordination component208for controlling message services210of a passive datacenter212.

As illustrated, the active coordination component202comprises two or more redundant DACs214that interface to the active datacenter services204to receive and process service queries. Similarly, the passive coordination component208can comprise two or more redundant DACs216that interface to the passive datacenter services210to receive and process service queries. Each of the active DACs214can include the server control information110, active request component102, and active response component112ofFIG. 1. Similarly, each of the passive DACs216can include the server control information120, passive request component126, and passive response component128ofFIG. 1.

Where multiple active DACs214are employed, the multiple active DACs214can intercommunicate to provide coordination management of the active services204to detect if a failure has occurred. Similarly, where multiple passive DACs216are employed, the multiple passive DACs216can intercommunicate to provide coordination management of the passive services210for the passive datacenter212.

The active datacenter206may be deactivated by an administrator or due to fault conditions such as a power failure. Under a fault condition, when power returns to the active datacenter206, the associated DACs214are reset to a “don't come to the online state”, and maintain the active services204in an offline (or down) state. The passive datacenter212is then activated by the administrator to provide high availability of the services210.

The DACs214collect status when a datacenter activation or deactivation occurs. The DACs214then expect the services204to post status on whether the services204were successful at performing the directed transition. If the services204failed at the transition, this failed transition information is useful, since the services204are likely to be in an undesired state. The status of the fault condition (e.g., in the active datacenter206) is posted to an associated coordination component (the active coordination component202). The status can include resulting state of a service, a failure code, and/or a failure message, for example. The services (e.g., active datacenter services204) of a datacenter can be configured to poll an associated coordination component (the active coordination component202) at regular time intervals for instructions to stop service, continue to provide service, or to start providing services.

Consider an example where the active datacenter206of messaging services204(e.g., a mailbox service) fails due to a power outage, no services will be are running, and the DACs214of the active coordination component202are also down. When power is restored to the active datacenter206, the services204can begin come up in an uncoordinated manner, and become active if left to come all the way up. Instead, the services204initiate a “Mommy may I” query to the DACs214. The service online state in the DACs214is reset to false (or stay offline) due to the power outage. Accordingly, the DACs214answer all queries with a “no” response to keep the services204in an offline state. Thus, the services204do not become active (or online) until the administrator interacts with the active coordination component202to bring the active datacenter206up by changing desired server state to “online”. Based on the administrator interaction, the active coordination component202will answer subsequent service queries with a “yes” response. The results of the action for the services204are then posted back as transition status to the DACs214.

In the passive datacenter212(presumably where the power outage did not occur) the administrator provides input to the passive coordination component208to the effect that “I want my passive datacenter online”. This effectively instructs the passive datacenter212to come online (active) while the active datacenter206is offline (passive).

FIG. 3illustrates an exemplary communications protocol between datacenter activation coordinators and datacenter servers and amongst the datacenter activation coordinators themselves. The protocol is described in the context of a passive system300where three passive (and redundant) DACs302are provided to manage the passive datacenter212of mailbox servers304. The figure shows the startup of DAC310. DAC310queries the other DACs308and312for their associated state. Each DAC308and310responds with its state; which may be “offline”, “online”, or “normal”. Normal allows active/passive cluster services to come online only when the two datacenters can communicate. In the “normal” case, the server makes its own decision about being online or offline. This is accomplished in the active/passive cluster environment by the server using its cluster logic to determine if it will be online or offline.

DAC310then assumes the state of the responses; which will agree. DAC310uses this state to respond to all mailbox server queries. The polling/response protocol is described in the context of server306querying DAC308. When a query Q (e.g., “May I come up?”) is received, the receiving DAC (e.g., DAC308) looks at the local state (stored in DAC308) and responds with an answer A (e.g., “Yes you may come up!”).

In other words, a DAC calculates desired state of the datacenter or a server using a specific algorithm which resets the desired state to offline when all DACs302go offline at the same time. The DAC (a server), being redundant, employs a community view of the stored datacenter state. When a given DAC server restarts, the DAC queries the other DAC servers to determine the desired state. An initializing DAC server starts with a desired state of “offline”. Only if the starting DAC server finds another DAC server reporting a desired state of “online” does the initializing DAC mark its desired state as “online”. By definition, taking all DAC servers down at the same time results in a datacenter failure. In this case, the administrator restores service after this type of event by intervening and marking the desired state as “normal”. The desired state can be persisted or not persisted.

As illustrated, the disclosed solution utilizes at least one redundantly deployed DAC in each datacenter to control the behavior of the messaging services in that datacenter. The DAC can control all messaging services in a datacenter, including services that run in an active/passive configuration. Each DAC controlled messaging service queries the local DAC(s) for the right to continue to or start providing service at a regular, configurable interval (e.g., approximately every thirty seconds). The DAC consults its stored state information and responds to the query such as the services may be instructed that they do not have the right, and thus, must shutdown. The DAC state information can be granular (e.g., per server or service) or datacenter wide (e.g., globally computed state). For example, the on/off state can be computed for the entire datacenter or individual servers.

When a state change is initiated the controlled services post completion status information to the local DAC. The completion status information can include, but is not limited to, resulting state of the service and/or server such as online, offline, normal, partially failed, failed, and/or in maintenance. This state can be distinct from the desired state. The state information can further include a failure code (if failed or partially failed), and failure message (if failed or partially failed).

In one implementation, the service and/or server does not update the status after a datacenter failure in that the mechanism is not deployed as an ongoing health monitoring system. In an alternative implementation, however, the service and/or server updates the status after a datacenter failure.

A service and/or server can locate its local DAC via a configured network identity. There can be multiple physical machines associated with the configured network identity. Alternatively, network load balancing can be employed. The DAC may or may not be a shared machine.

FIG. 4illustrates a computer-implemented method of managing a datacenter. At400, control information at redundant activation coordinators of a datacenter is resolved due to a failure of the datacenter. At402, queries (e.g., polling) are received from application servers of the datacenter for service status based on the control information. At404, answers to the queries are sent that control the services to prevent a split-brain scenario between the datacenter and one or more passive datacenters.

The method can further comprise changing state of the control information in control tables of the activation coordinators to activate or deactivate the application servers. The control information can be resolved, by an activation coordinator in startup mode querying one or more other activation coordinators of the datacenter for state information and determining the control information based on the state information. The control information is manipulated manually to activate or deactivate the application servers. The redundant activation coordinators share state upon startup, each coordinator receives the queries, processes the queries, and sends the answers to the application servers associated with the queries.

The method can further comprise posting completion status information of the application servers to the redundant activation coordinators. The completion status information can include resulting state of the application servers, a failure code, and a failure message. The method can further comprise maintaining a passive datacenter in a passive mode by sending answers to queries from application servers of the passive datacenter that keep the application servers of the passive datacenter deactivated.

FIG. 5illustrates a method of managing application services based on a active datacenter failure. At500, during normal operation, active datacenter state is stored in the redundant instances of active datacenter activation coordinators. At502, a DAC service queries other DAC instances on startup of the DACs. The DAC startup process can be due to individual maintenance on the single DAC or due to a datacenter-wide failure. At504, if all DAC instances are starting, the single DAC service starting up assumes a system-wide power or other failure. Alternatively, if all of the other DAC instances are running, the assumption can be that no failure has occurred and to simply assume the state of the other DAC instances. Note also that during a datacenter-wide failure that the other DAC instances are also starting up and querying the other DAC(s) for state information. Accordingly, at506, since the other DAC instances are also starting up, all the DACs operate to block activation of the application (app) services of the datacenter to prevent a split-brain scenario with another datacenter. At508, recovery is initiated from a datacenter-wide failure by updating the state to “online”. At510, each of the local servers subsequently sends a request to the activation coordinator for the desired state. At512, a response from the activation coordinator triggers a state change of the servers. At514, the result of the state change is posted back to the DAC. However, if all instances of the DAC services are not starting, flow is from506to516to send notification to an administrator, for example, and process the desired state accordingly.

As used in this application, the terms “component” and “system” are intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to being, a process running on a processor, a processor, a hard disk drive, multiple storage drives (of optical and/or magnetic storage medium), an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a server and the server can be a component. One or more components can reside within a process and/or thread of execution, and a component can be localized on one computer and/or distributed between two or more computers. The word “exemplary” may be used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs.

Referring now toFIG. 6, there is illustrated a block diagram of a computing system600operable to support centralized activation/deactivation control of datacenters in accordance with the disclosed architecture to prevent a split-brain scenario. In order to provide additional context for various aspects thereof,FIG. 6and the following discussion are intended to provide a brief, general description of a suitable computing system600in which the various aspects can be implemented. While the description above is in the general context of computer-executable instructions that may run on one or more computers, those skilled in the art will recognize that a novel embodiment also can be implemented in combination with other program modules and/or as a combination of hardware and software.

With reference again toFIG. 6, the exemplary computing system600for implementing various aspects includes a computer602having a processing unit604, a system memory606and a system bus608. The system bus608provides an interface for system components including, but not limited to, the system memory606to the processing unit604. The processing unit604can be any of various commercially available processors. Dual microprocessors and other multi-processor architectures may also be employed as the processing unit604.

The system bus608can be any of several types of bus structure that may further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. The system memory606can include non-volatile memory (NON-VOL)610and/or volatile memory612(e.g., random access memory (RAM)). A basic input/output system (BIOS) can be stored in the non-volatile memory610(e.g., ROM, EPROM, EEPROM, etc.), which BIOS are the basic routines that help to transfer information between elements within the computer602, such as during start-up. The volatile memory612can also include a high-speed RAM such as static RAM for caching data.

The computer602further includes an internal hard disk drive (HDD)614(e.g., EIDE, SATA), which internal HDD614may also be configured for external use in a suitable chassis, a magnetic floppy disk drive (FDD)616, (e.g., to read from or write to a removable diskette618) and an optical disk drive620, (e.g., reading a CD-ROM disk622or, to read from or write to other high capacity optical media such as a DVD). The HDD614, FDD616and optical disk drive620can be connected to the system bus608by a HDD interface624, an FDD interface626and an optical drive interface628, respectively. The HDD interface624for external drive implementations can include at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

A number of program modules can be stored in the drives and volatile memory612, including an operating system630, one or more application programs632, other program modules634, and program data636. Where the computer602is employed as a coordination component (e.g., coordination component116), the one or more application programs632, other program modules634, and program data636can include the response component112, the server control information110, the request component102, the active coordination component202, and the active DACs214, for example. This applies similarly to the passive coordination components (118,208and302).

One or more of these components can be embodied as services/servers. For example, the DACs and coordination components can be services (or instances) operating to perform the desired functions as described supra. The computer602can be the physical system that supports the components described herein. A datacenter (e.g., datacenter102, passive datacenter112, etc.) can comprise multiple computers602that host one or more operational services (e.g., services106which may or may not fault.

All or portions of the operating system, applications, modules, and/or data can also be cached in the volatile memory612. It is to be appreciated that the disclosed architecture can be implemented with various commercially available operating systems or combinations of operating systems.

A user can enter commands and information into the computer602through one or more wire/wireless input devices, for example, a keyboard638and a pointing device, such as a mouse640. Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, touch screen, or the like. These and other input devices are often connected to the processing unit604through an input device interface642that is coupled to the system bus608, but can be connected by other interfaces such as a parallel port, IEEE 1394 serial port, a game port, a USB port, an IR interface, etc.

A monitor644or other type of display device is also connected to the system bus608via an interface, such as a video adaptor646. In addition to the monitor644, a computer typically includes other peripheral output devices (not shown), such as speakers, printers, etc.

The computer602may operate in a networked environment using logical connections via wire and/or wireless communications to one or more remote computers, such as a remote computer(s)648. The remote computer(s)648can be a workstation, a server computer, a router, a personal computer, portable computer, microprocessor-based entertainment appliance, a peer device or other common network node, and typically includes many or all of the elements described relative to the computer602, although, for purposes of brevity, only a memory/storage device650is illustrated. The logical connections depicted include wire/wireless connectivity to a local area network (LAN)652and/or larger networks, for example, a wide area network (WAN)654. Such LAN and WAN networking environments are commonplace in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which may connect to a global communications network, for example, the Internet.

When used in a LAN networking environment, the computer602is connected to the LAN652through a wire and/or wireless communication network interface or adaptor656. The adaptor656can facilitate wire and/or wireless communications to the LAN652, which may also include a wireless access point disposed thereon for communicating with the wireless functionality of the adaptor656.

When used in a WAN networking environment, the computer602can include a modem658, or is connected to a communications server on the WAN654, or has other means for establishing communications over the WAN654, such as by way of the Internet. The modem658, which can be internal or external and a wire and/or wireless device, is connected to the system bus608via the input device interface642. In a networked environment, program modules depicted relative to the computer602, or portions thereof, can be stored in the remote memory/storage device650. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers can be used.