Forced management module failover by BMC impeachment consensus

A computer-implemented method, system and computer program product for managing failover of Management Modules (MMs) in a blade chassis are presented. Each server blade in the blade chassis evaluates a performance of a primary MM. If a threshold number of server blades determine that the primary MM is not meeting pre-determined minimum performance standards, then a secondary MM impeaches the primary MM and takes over the management of the server blades.

BACKGROUND OF THE INVENTION

1. Technical Field

The present disclosure relates to the field of computers, and specifically to blade servers. Still more particularly, the present disclosure relates to managing Management Modules (MMs) that support blades in a blade server chassis.

2. Description of the Related Art

While early computer architectures utilized stand-alone single computers, often referenced as Personal Computers (PCs), more powerful modern computer systems often use multiple computers that are coupled together in a common center. An exemplary common center is known as a blade center, which utilizes multiple blades that are coupled by a common backbone in a blade chassis. Each blade is a pluggable board that comprises at least one processor, on-board memory, and an Input/Output (I/O) interface. The multiple blades are capable of communicating with one another, as well as sharing common resources, such as storage devices, monitors, input devices (keyboard, mouse), etc.

Current blade chassis (also known as blade centers) provide a mechanism whereby the Management Module (MM) can failover to a redundant MM under certain conditions. There is a deficiency in the art, however, in that scenarios in which this occurs is limited. There exist times wherein a standby MM cannot determine that it needs to take over for a failed MM. For example, the standby MM could have no idea that the primary MM was too busy or is not properly servicing interrupts sent from the blade service processors.

SUMMARY OF THE INVENTION

To address the above described issue, a computer-implemented method, system and computer program product for managing failover of Management Modules (MMs) in a blade chassis are presented. Each server blade in the blade chassis evaluates a performance of a primary MM. If a threshold number of server blades determine that the primary MM is not meeting pre-determined minimum performance standards, then a secondary MM impeaches the primary MM and takes over the management of the server blades.

The above, as well as additional purposes, features, and advantages of the present invention will become apparent in the following detailed written description.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

With reference now to the figures, and in particular toFIG. 1, there is depicted a block diagram of an exemplary blade102, which may be utilized by the present invention. Blade102is a component of a server blade chassis (depicted below inFIG. 2as blade chassis202). Blade chassis202is usually utilized as a server; thus, blade102may be referenced as a server blade. Note that some or all of the exemplary architecture shown for blade102may be utilized by software deploying server150and/or other blades124, which may be within a same blade chassis as blade102.

Blade102includes a processor unit104that is coupled to a system bus106. Processor unit104may utilize one or more processors, each of which has one or more processor cores. A video adapter108, which drives/supports a display110, is also coupled to system bus106via a chassis interface131to a chassis backbone218(described in greater detail below). In one embodiment, a switch107couples the video adapter108to the system bus106. Alternatively, the switch107may couple the video adapter108to the display110. In either embodiment, the switch107is a switch, preferably mechanical, that allows the display110to be coupled to the system bus106, and thus to be functional, only upon execution of instructions (e.g., Management Module Control Program—MMCP148described below) that perform the method described herein. This switching causes a substantive transformation of the blade102from a system in which pending steps and/or results of the herein described method are NOT displayed, into a system in which these results ARE displayed.

System bus106is coupled via a bus bridge112to an Input/Output (I/O) bus114. An I/O interface116is coupled to I/O bus114. I/O interface116affords communication with various I/O devices either directly or via the chassis interface131, which is hardware and/or software that allows the blade102to be coupled to chassis backbone218in a blade chassis (described in further detail inFIG. 2). Once coupled to the chassis backbone218, the blade102is able to communicate with other devices in addition to the display110, including a keyboard118, a mouse120, a media tray122(which may include storage devices such as CD-ROM drives, multi-media interfaces, etc.), other blade(s)124that are within a blade chassis, and (if a VHDL chip137is not utilized in a manner described below), USB port(s)126. Note that while other blade(s)124are shown as being coupled to blade102via the chassis interface131, in one embodiment these other blade(s)124can be coupled to blade102via network128, particularly if network128is a Local Area Network (LAN) within a blade center. While the format of the ports connected to I/O interface116may be any known to those skilled in the art of computer architecture, in a preferred embodiment some or all of these ports are Universal Serial Bus (USB) ports. Also coupled to I/O bus114is a Baseboard Management Controller (BMC)210, which is discussed further below.

As depicted, blade102is able to communicate with a software deploying server150and, in one embodiment, with other blade(s)124within a blade chassis, via network128using a network interface130, which is coupled to system bus106. Network128may be an external network such as the Internet, or an internal network such as an Ethernet or a Virtual Private Network (VPN).

Application programs144in blade102's system memory (as well as software deploying server150's system memory) also include a Management Module Control Program (MMCP)148. MMCP148includes code for implementing the processes described below, including those described inFIGS. 2-3. In one embodiment, blade102is able to download MMCP148from software deploying server150, including in an on-demand basis. Note further that, in one embodiment of the present invention, software deploying server150performs all of the functions associated with the present invention (including execution of MMCP148), thus freeing blade102from having to use its own internal computing resources to execute MMCP148.

Also stored in system memory136is a VHDL (VHSIC Hardware Description Language) program139. VHDL is an exemplary design-entry language for Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), and other similar electronic devices. In one embodiment, execution of instructions from MMCP148causes VHDL program139to configure VHDL chip137, which may be an FPGA, ASIC, etc. This programming of VHDL chip137causes a substantial transformation of the architecture of blade102, wherein (assuming that USB port(s)126are NOT coupled to I/O interface116) USB port(s)126are now selectively coupled to system bus106via VHDL chip137.

In another embodiment of the present invention, execution of instructions from MMCP148results in a utilization of VHDL program139to program a VHDL emulation chip151. VHDL emulation chip151may incorporate a similar architecture as described above for VHDL chip137. Once MMCP148and VHDL program139program VHDL emulation chip151, VHDL emulation chip151performs, as hardware, some or all functions described by one or more executions of some or all of the instructions found in MMCP148. That is, the VHDL emulation chip151is a hardware emulation of some or all of the software instructions found in MMCP148. In one embodiment, VHDL emulation chip151is a Programmable Read Only Memory (PROM) that, once burned in accordance with instructions from MMCP148and VHDL program139, is permanently transformed into a new circuitry that performs the functions of elements204,206, and/or212shown below inFIG. 2. Thus, VHDL emulation chip151is also properly viewed as a machine that is under the control of blade102. Note that while VHDL emulation chip151is depicted as being a different entity that is separate from blade102, in another embodiment VHDL emulation chip151may be an integral part of blade102.

The hardware elements depicted in blade102are not intended to be exhaustive, but rather are representative to highlight essential components required by the present invention. For instance, blade102may include alternate memory storage devices such as magnetic cassettes, Digital Versatile Disks (DVDs), Bernoulli cartridges, and the like. These and other variations are intended to be within the spirit and scope of the present invention.

Referring now toFIG. 2, an exemplary blade chassis202, in which the present invention may be implemented, is presented. Within blade chassis202are a plurality of blades208a-n,where “n” is any integer, but is preferably 14. The blades208a-n(each referenced individually as a set/subset without specificity as208) are coupled to a chassis backbone218, which provides mechanical and logical connections (e.g., data and control signal interchange) among the blades208a-n.Associated with, and preferably mounted on each blade208, is a separate Baseboard Management Controller (BMC)210. That is, in a preferred embodiment, each blade208has its own unique BMC210. In an alternate embodiment, multiple blades208may share a same BMC210. In either architecture, BMC210is a specialized microcontroller that monitors the overall health and environment of a blade208. This monitoring includes, but is not limited to, controlling cooling fans214, adjusting power supply216for one or more blades208, checking on an Operating Status (OS) within a particular blade208, and then sending warning and alerts regarding anomalies to such monitored activities to an administrator (not shown). In accordance with the present invention, the BMC210also monitors a performance of a primary Management Module (MM)204, to which a particular blade208(e.g., blade208n) is presently coupled.

Primary MM204(and likewise secondary MM206) includes a processor (not shown) for controlling Input/Output (I/O) functions of specific blades208, interfacing a specific blade with a network (e.g., network128shown inFIG. 1), and allocating jobs and data to specific blades208.

As stated earlier, BMC210is able to monitor the activities of primary MM204, particularly as those activities directly relate to a specific blade208, such as the exemplary blade208ndepicted inFIG. 2. A pre-determined threshold for activity performance levels may be set, such that if the BMC210determines that these thresholds are not being met, specific actions can be taken. In the present invention, these specific actions include sending a message to the secondary MM206, letting the secondary MM206know that the primary MM204is not performing up to required levels. The secondary MM206can then tally such messages from different BMCs210on other blades208, in order to decide if the primary MM204needs to be impeached (disconnected by the performance-based switching logic212) and replaced by the secondary MM206. In an alternate embodiment, the potential of a “rogue” secondary MM206overthrowing the primary MM204can be reduced by assigning the authority and responsibility of impeaching the primary MM204to the performance-based switching logic212, which may include processing logic (not shown) for performing this task. For example, the secondary MM206may be defective (e.g., from a virus, a mechanical failure, etc.), leading it to overthrow the primary MM204, for no reason. Allowing the performance-based switching logic212to control failover avoids, or at least minimizes, this problem.

Note that in a preferred embodiment, each of the blades208a-nhas its own unique BMC210. Alternatively, multiple blades208may share a single BMC210. Note also that components within blade chassis (e.g., blades208a,baseboard management controller210, performance-based switching logic212, and primary/secondary management modules204and206) may be coupled together by a RS485 communication path (not shown). Before failover, secondary MM206has an active network connection to an internal (secondary) network (also not shown), but has no access to the RS485 communication path until the secondary MM206takes over as the primary MM. Service processor alerts may be sent using the RS485 network to the primary MM, which may be either the primary MM204or the secondary MM206(after the secondary MM206is failed-over to).

With reference now toFIG. 3, a high level flow-chart of exemplary steps taken to manage the failover of management modules in a blade chassis is presented. After initiator block302, a primary Management Module (MM) is initialized to supervise server blade operations in one or more server blades in a blade chassis (block304). This initialization includes the physical and logical coupling (e.g., via an RS485 network and a backbone) of the server blades to the primary MM. In a preferred embodiment, this coupling is via a performance-based switching logic that selectively couples one or more of the server blades to either a primary MM or a secondary MM. In one embodiment, this selective coupling is under the direction and management of the secondary MM in a manner described below.

As described in query block306, a Baseboard Management Controller (BMC) in each of the server blades determines whether the primary MM is meeting pre-determined minimum performance requirements. These pre-determined minimum performance standards may be based on how rapidly and/or efficiently and/or correctly the primary MM distributes work among the server blades or to a specific server blade, how long it takes the primary MM to respond to a request from a server blade for service (e.g., providing an interface and connection to another server blade or a network, timely providing data and/or instructions to a requesting server blade, etc.), etc. The BMC's evaluation may result in a score that indicates how well the primary MM is performing, including whether the primary MM has failed to meet a set of cumulative pre-determined minimum performance requirements. A tally of different minimum performance standards for different services provided by the primary MM can then be used to assign a pass/fail rating for the primary MM, as judged by a particular BMC/blade.

If the BMC determines that the primary MM is not meeting the pre-determined minimum performance requirements (again query block306), then the BMC from a particular server blade sends a complaint signal to the secondary MM (block308). This complaint may have the authority to force the secondary MM to instruct a performance-based switching logic to decouple the primary MM and to failover to the secondary MM. In a preferred embodiment, however, this complaint signal is merely a “vote,” which is added to other votes from other BMCs in other server blades. If the weight of the vote(s) is sufficient (query block310), then the secondary MM determines that a failover threshold has been reached, and the secondary MM instructs the performance-based switching logic to decouple the primary MM and to failover to the secondary MM (i.e., couple the secondary MM to the server blades). Note that this failover may cause all of the server blades to now be coupled to the secondary MM, or alternatively, may cause only some of the server blades to be coupled to the secondary MM while other server blades remain coupled to the primary MM. The decision as to which (or all) of the server blades are coupled to one or the other MM may be based on the secondary MM's evaluation of the performance of the primary MM (based on the complaint signals and/or votes from the BMCs) and the capacity and capabilities of the secondary MM (of which the secondary MM is self-aware). Note that the votes and complaint signals may be a same signal, or preferably, are different messages. That is, in a preferred embodiment, a vote is merely a “go/no go” message to the secondary MM to impeach/evict/failover the primary MM, while the complaint signal is a finer-grained message that describes in more detail what deficiencies in the primary MM are being perceived by a particular BMC in a particular server blade. Note also that the complaint signal may be integral to a primary MM performance score set by the BMC. Thus, the BMC can dynamically and autonomously adjust what this score should be in order to be “acceptable.” That is, the BMC can dynamically adjust its benchmarks for the primary MM such that a vote and/or complaint signal may be dependent on environmental issues (e.g., how much total traffic from all server blades is being handled by the primary MM), time of day (e.g., a BMC may be more lenient in its grading during the historically busy middle of a day compared to a historically slower middle of the night), etc. The BMC is therefore able to dynamically adjust its criteria for whether the primary MM should be impeached. This is accomplished by adjustment logic (i.e., a processor) within the BMC.

Returning to query block310, if complaint signals and/or votes from one or more BMCs exceed the minimum combined performance requirement that is required of the primary MM, then the secondary MM stages a coup (block312), in which the primary MM is ousted, and the secondary MM takes over the MM role for the complaining server blades. Note again that this reallocation is preferably an “all-or-none” scenario, in which the secondary MM takes over all of the duties of the primary MM, and thus supervises all of the server blades in the blade chassis. Alternatively, the failover may be a partial failover, in which the primary and secondary MM share management responsibilities for the server chassis. Note also that once the failover occurs, the primary MM may now become the backup (secondary) MM for failover purposes.

Note also, as described in block312, that work can be reallocated if there is a failover or a failed failover attempt. That is, if there are enough complaint signals and/or impeachment votes from the BMCs, then the secondary MM may decide that it is unable, even with the primary MM's help, to handle all of the demands of the server blades and their BMCs. In this scenario, the secondary MM will call another server chassis, and will transfer some or all of the currently pending jobs in the local server chassis to the other server chassis. The process ends at terminator block314.

As described in one embodiment herein, the present invention uses an internal network as a communication path for the blade service processors to complain about (vote to impeach) the primary MM. These votes are targeted at the standby (secondary) MM, which collects the votes and performs periodic tallies of the votes to determine if an impeachment (failover) is required. Once a configurable threshold of votes is collected, then the standby MM resets the primary MM and takes over as primary. The previously primary MM then goes into a standby mode and the service processors are updated with the new MM (primary and standby) IP information and the voting process begins again.

As described above, in one scenario a Management Module is flooded with tasks and is not adequately handling critical requests from BMCs onboard the individual blades. The management module may be unaware that it is dropping packets or failing, because it is busy or broken. BMCs in one or more server blades are empowered to raise a signal to indicate that the BMC(s) believe that the current Management Module is unfit to continue its duties and request a failover. BMCs raise this signal according to a configured or hard-coded (pre-determined) failover policy. The redundant (backup/secondary) management module is implemented to receive, or monitor for, these signals from the various BMCs in the system. If a threshold number of BMCs have raised a signal that they are in favor of a failover, the redundant management module is empowered to assume primary management device responsibility and notify all BMCs that this failover has occurred, whereupon all BMCs may lower their overthrow signals and continue normal operation. Once a BMC raises its overthrow signal, it can lower the threshold for such a signal on its own before a failover takes place, thus affording time for an improvement in behavior by the management module to be observed (i.e. give the primary MM a “second chance”). In one embodiment, a primary management module, upon seeing overthrow flags raised, takes actions to attempt to improve its quality of service, and/or to send a predictive failure/failover alert to an administrator.

In another embodiment, the BMCs agree that a failover of a more significant magnitude needs to occur. For example, if the Heating Ventilation Air Conditioning (HVAC) systems in a server room give out, thereby frying the management modules, the BMCs are enabled to signal for a datacenter-level failover, which enables an administrator of some other agent to shift the datacenter workload to another site in absence of a Management Module initiated directive.

It should be understood that at least some aspects of the present invention may alternatively be implemented in a computer-readable medium that contains a program product. Programs defining functions of the present invention can be delivered to a data storage system or a computer system via a variety of tangible signal-bearing media, which include, without limitation, non-writable storage media (e.g., CD-ROM), and writable storage media (e.g., hard disk drive, read/write CD ROM, optical media). It should be understood, therefore, that such storage media when encoded with computer readable instructions that direct method functions in the present invention, represent alternative embodiments of the present invention. Further, it is understood that the present invention may be implemented by a system having means in the form of hardware, software, or a combination of software and hardware as described herein or their equivalent.

Note further that any methods described in the present disclosure may be implemented through the use of a VHDL (VHSIC Hardware Description Language) program and a VHDL chip. VHDL is an exemplary design-entry language for Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), and other similar electronic devices. Thus, any software-implemented method described herein may be emulated by a hardware-based VHDL program, when is then applied to a VHDL chip, such as a FPGA. Applying the VHDL instructions to the VHDL chip not only causes a physical transformation of the VHDL chip, but such VHDL instruction application can also cause a specifically loaded VHDL chip to be newly coupled (physically and/or logically) to other hardware within a computer system, thus causing a additional physical transformation of the computer system.