Distributed baseboard management controller for multiple devices on server boards

A server board includes first and second devices. A first service processor of the first device operates as a master baseboard management controller of the server board, and monitors a communication channel for alive messages from a plurality service processors. A second service processor operates as a secondary baseboard management controller, and sets a second timer to a first value. In response to a determination that the second timer has expired based on a first value: the second service processor to start a switchover process, and to set the second timer to a second value based on an alive message period. In response to a primary alive message not being received from the first service processor prior to the second timer expiring based on the second value, the second service processor to reset first service processor and to operate as the master baseboard management controller.

BACKGROUND

Field of the Disclosure

Server boards can be used in a data center for computing, networking or storage applications. These server boards typically have a single centralized baseboard management controller (BMC). The BMC can utilize protocols, such as those put further in the Intelligent Platform Management Interface (IPMI) standard, to monitor and control other components of the server board. The BMC can monitor and control the components to help ensure that they are available, to perform load balancing, and to collect statistics from these components. The BMC can utilize a dedicated memory, dedicated communication ports, and dedicated power supply circuitry on the server board. As such, the cost of the server board increases to include the BMC functionality on top of the basic server board components. Additionally, failure of the single BMC can cause the entire server board to fail.

DETAILED DESCRIPTION OF THE DRAWINGS

A server board includes multiple devices each having one or more processors, multiple processor memories corresponding to the devices, and a shared system memory in accordance with at least one embodiment of the present disclosure. Each device includes a respective service processor, multiple general purpose processors, and a memory controller. Each of the service processors can monitor the components on its corresponding device. The service processors operate together to implement a distributed baseboard management controller (BMC). According to an embodiment, one of the service processors is a primary BMC that coordinates implementing BMC functionality for the server board, while each of the other service processors are slave BMCs that operate in a defined manner to support the master BMC.

One or more of the slave BMCs can be a secondary master that, in addition to implementing slave operations, monitors the health of the primary BMC. Therefore, the server board includes a distributed BMC architecture that allows one service processor to operate as a master BMC, another service processor to operate as a slave and a secondary BMC, and the remaining service processors to operate as slave BMCs. The slave service processors can provide alive messages to a communication bus that is indicative of their corresponding devices operating normally. The master BMC can detect the presence, or lack, of the alive messages at the communication bus from the other service processors. The master BMC service processor can manage expiration timer values corresponding to each of the other service processors to monitor whether valid alive messages are received from each of the other service processors within an expected amount of time, thus tracking the status of the other service processors. The master BMC can implement appropriate remedial actions with respect to the other service processors when a possible failure is detected.

In a further embodiment, while the master BMC service processor monitors the other BMC service processors, the secondary BMC service processor monitors the master BMC service processor to determine whether the master BMC service processor has failed. The secondary BMC can implement appropriate remedial actions with respect to the master service processors when a possible failure is detected, or even switch-over to operate as the master BMC. By distributing the BMC functionality across the service processors of the server board, there is no need for an additional device that implements BMC functionality to monitor the devices of the server board.

FIGS. 1-5illustrate a server board100in accordance with at least one embodiment of the present disclosure. The server board100includes a plurality of devices102,104,106, and108(devices102-108), multiple processor memories110,112,114, and116(memories110-116), and a shared system memory118. In an embodiment, devices102-108are presumed to be have homogenous architectures, and as such each have similar features as indicated by the unit values of their corresponding reference numbers. For example, each of the devices102-108can include a corresponding one of service processors120,130,140, and150(service processors120-150), a corresponding one of general purpose processors122,132,142, and152(general purpose processors122-152), a corresponding one of memory controllers124,134,144, and154(memory controllers124-154). In different embodiments, the service processors120-150can be processors of the same or different type architecture as the general purpose processors122-152. Each of the service processors120-150can include baseboard management controllers (BMC)126,136,146, or156(BMCs126-156) that implement baseboard management controller functionality by operating together to implement a distributed BMC. Each of the service processors120-150can include a corresponding timer128,138,148, or158. The timers128,138,148, and158(timers128-158) have been illustrated as part of the respective service processors120,130,140, and150. However, the timers128-158may be external to the respective service processors120-150without varying from the scope of this disclosure.

Each of the processor memories110-116include a storage location160,162,164, or166(storage locations160-166) that is used to store timer information used to implement a corresponding watch dog (WD) timer. The shared system memory118includes a storage location168to store WD timer information for each of the service processors120-150storage location168. The plurality of devices102-108can utilize their respective memory controllers124-154to communicate with the shared system memory118and with their corresponding processor memories110-116. The devices102-108can communicate with one another via a communication bus170. In an embodiment, the service processors102-108can communicate in compliance with the Intelligent Platform Management Interface (IPMI) standard.

During operation, the service processors120-150can provide on-chip debug processes and implement various WD timers to monitor on-chip processing by storing appropriate values in the storage locations160-166of the respective processor memories110-116. For example, a separate time value can be stored at location WD160for each one of the general purpose processors122to monitor its health. Thus, the service processor120can monitor alive messages from each of the general purpose processors122, wherein an alive message can indicate that a corresponding general purpose processor122is working properly. Upon receiving an alive message from one of the general purpose processors122, the service processor120can update a watch dog timer value in the storage location160that corresponds to the general purpose processor122that provided the alive message. Should a time implementing this watch dog timer value expire before receiving a next alive indicator from the same general purpose processor, it can be an indication that there is a problem with the particular processor.

The BMC126of service processor120can be implemented by executing code at an instruction processor of service processor120that implements master BMC functionality for the server board100. The service processor130of device104can execute the code to implement slave BMC functionality and secondary BMC functionality of device104. The service processors140and150of devices106and108, respectively, can execute the code to implement slave BMC functionality for their respective devices, as described in greater detail below with respect toFIGS. 2-5below.

In an embodiment, each of the service processors120,130,140, and150can operate to implement a portion of a distributed BMC architecture with service processor120as the master BMC, service processor130as the secondary BMC, and service processors140and150as slave BMCs. For brevity and clarity, only four devices have been shown within server board100. However, server board100may include additional or fewer devices within without varying from the scope of this disclosure.

Referring toFIG. 2, each of the service processors130,140, and150can implement BMC slave functionality by periodically writing an alive message/indicator to the communication bus170to indicate it is operating properly. The service processor120operates as the master BMC to monitor the communication bus170for alive messages from the service processors130,140, and150. As the master BMC, the service processor120can have ownership of the storage location168of the shared system memory118, and can store respective expiration timer values for each one of the service processors120,130,140, and150(e.g., WD120, WD130, WD140, and WD150) in the storage location168. A particular expiration timer value can be updated by the master BMC, e.g., the service processor120, in response to an alive message being received from its corresponding service processor. For example, expiration timer value WD150can be updated each time service processor120detects an alive message from service processor150. The expiration timer values WD130, WD140, and WD150, can be utilized by the service processor120to track the status of the other service processors130,140, and150as discussed in greater detail below.

The service processor150can provide an alive message202, represented inFIG. 2by arrow202, to the service processor120via the communication bus170. In response to receiving the alive message202, the service processor120can determine that the service processor150is alive, e.g., working properly. In an embodiment, the determination that a service processor, such as service processor150, is working properly in turn means that the service processor150has determined other components of the device108, e.g., general purpose processors152, are also working properly.

In response to determining that service processor150is alive, the service processor120can write a new expiration timer value, WD150, for the service processor150in the watch dog expiration timer's storage location168, as represented by the arrow204in response to receiving the alive message202. In an embodiment, the new expiration timer value can be set to a time stamp of device102(TS102) plus a failure indication period (FIP), such that WD150=TS102+FIP. In an embodiment, the time stamp can be a clock time of device102when the alive message202is received at the service processor120, and the failure indication period can be a preset length of time between when a single slave BMC is expected to provide two successive alive messages.

It will be appreciated, that the service processor120can maintain the value of timer128to be the value of the next to expire expiration timer value in the watch dog expiration timer storage location168, assuming no other timers are being implemented using timer128. In an embodiment, the next to expire expiration timer value will generally not be the expiration timer value that the service processor120updated most recently, e.g., WD150, because this updated value will most likely be later in time than the expiration timer value for either service processor130or140. Therefore, the next to expire expiration timer value will correspond to either service processor130or service processor140. For example, the next to expire expiration timer value can correspond to the service processor140, WD140. In an embodiment, the value set in the timer128can be a specific time value of the clock for the service processor120, and the timer128can expire when that point in time is reached.

In an embodiment, each of the other service processors, e.g., service processors130and140, also operates as a slave BMC, allowing the service processor120to monitor each device of the server board100having a slave BMC in a similar way as described above with respect to device108and service processor150. In an embodiment, the service processor130of device104additionally operates as a secondary master BMC device that can monitor alive messages from service processor120, in case of failure of device102, in a similar fashion as service processor120monitors the devices104-108, as will be discussed in greater detail below.

Referring now toFIG. 3, the service processor120can detect that the timer128has expired. In an embodiment, the timer128was previously set to the next to expire expiration timer value, presumed to be WD140of the service processor140. In this embodiment, the expiration of the timer128can cause the service processor120to make the determination that the service processor140, or one of the other components within device106, has failed or may have failed. In response, the service processor120can attempt to implement remedial operations, such as communicating with service processor140, providing a reset signal302to service processor140, which in turn can cause a service processor140to implement a reset operation on one or more of the components within the device106. E.g., the service processor140, the general purpose processors142, the memory controller144, and the timer148.

After providing the reset signal302, the service processor120can update the expiration timer value corresponding to the service processor140, WD140, in the storage location168via an update signal304. In an embodiment, the updated expiration timer value, WD140, can be the time stamp, TS120, of when the reset signal302was sent to the service processor140plus a boot period, BP, (e.g., WD140=TS120+BP). In an embodiment, the boot period can be the length of time for the components of a device to complete a boot operation after being reset. The service processor120can then set the timer128to a value of the next to expire expiration timer value in the watch dog expiration timer's storage location168. In an embodiment, the next to expire expiration timer value will not be the expiration timer value corresponding to service processor140, WD140, based on this expiration timer value being recently updated.

In different embodiments, each of the devices102-108can maintain different clocks or can have the same clock. In an embodiment where the devices102-108have the same clock, the service processors120and130(as will be discussed in greater detail below) can read and write expiration values to the storage location168without a time conversion needed. In an embodiment where the devices102-108maintain different clocks, the service processor130converts the expiration timer values in the storage location168to match the clock of the device104.

Referring now toFIG. 4, the service processor120can provide an alive message402, referred to as a primary alive message, to the service processor130. In an embodiment, the service processor120can send the primary alive message at regular intervals, and include a time stamp of the current time of in the clock of the service processor120being used to maintain the values stored in expiration timer168. The service processor120can also update the expiration timer value for itself, WD120, within the storage location168, via an update message404. In an embodiment, the new expiration timer value, WD120, can be set to the time stamp, TS120, plus the failure indication period, FIP.

The service processor130can utilize the time stamp in the primary alive message402to calculate a difference between the clock of the service processor120and its own clock, e.g., a clock of the service processor130. Thus, the service processor130can calculate the difference between the clock of device102and the clock of device104. The service processor130can then store this difference for later use to convert expiration timer values stored, by service processor120, in the storage location168to the clock of device104.

According to an embodiment, service processor120will send an alive message to the secondary service processor130each time service processor120receives an alive signal. Upon receiving an alive indicator from service processor120, the secondary service processor130can determine when the next alive indicator is expected from the primary service processor120by reading the next to expire value from storage location168. For example, after receiving an alive signal from the primary service processor120, the secondary service processor130can send a retrieve message406to the storage location168to retrieve the timer value stored at location168that represents the next to expire expiration timer value, as maintained by the service processor120. The service processor130can then utilize the clock difference to convert the retrieved next to expire expiration timer value to the clock of the device104, which is then used to set the timer138to equal the adjusted next to expire expiration timer value. Should this timer expire it may be an indication that device102has failed, that device106has failed, or that devices108has failed. In one embodiment, the timer138can be set to a different value than the timer128, such that the service processor120can have time to reset a failed device prior to the timer138expiring. In this embodiment, when the timer138expires the service processor130determines whether value of timer matches any value in storage location168, if so this match may indicate that service processor120has failed.

Referring now toFIG. 5, the service processor130, operating as the secondary BMC, can detect that the service processor120may have failed, by determining that the timer138has expired. In an embodiment, upon the timer138expiring, e.g., the timer value stored in the timer138matching the value of the clock of the service processor130, the service processor130will retrieve the next to expire expiration timer value from the storage location168via a retrieve signal502. The service processor130can then compare the next to expire expiration timer value to the current value in the timer138. In an embodiment, the clock of the service processor120and the clock of the service processor130can differ, such that the service processor130can utilize the saved clock difference to convert the next to expire expiration timer value from the clock of the service processor120to the clock of the service processor130prior to the comparison being performed.

The comparison of the current value of the timer138to the next to expire expiration timer value can result in two outcomes in the service processor130. For example, if the next to expire expiration timer value is different from the current value of the timer138, the service processor130can determine that all of the service processors, e.g., service processors120,140, and150, are working properly, and the service processor130can set the timer138to the next to expire expiration timer value and continue operating as the secondary BMC as described below. However, if the next to expire expiration timer value is the same as the current value of the timer138, the secondary processor130can determine that a failure has occurred in the server board100, e.g., at one of the service processors120,140, or150, as will be described below, by virtue of service processor120not updating the next to expire expiration value.

In an embodiment, the next to expire expiration timer value does not change within the storage location168if the service processor120does not provide an updated expiration timer value to the storage location168. One situation that would cause the next to expire expiration timer value not to have changed, is that the service processor (e.g., service processor140) corresponding to the next to expire expiration value has failed and the service processor120is in the process of resetting the service processor140, such that the service processor120has not yet updated the expiration timer value for the service processor140. Another situation that would cause the next to expire expiration timer value not to have changed, is that the service processor120, e.g., the master BMC, has failed. Thus, the expiration of the timer138can cause the service processor130to retrieve the next to expire expiration timer value from the storage location168via the retrieve signal502, then to convert the next to expire expiration timer value to the clock of the service processor130, and compare the next to expire expiration timer value to the current value of the timer to determine if the next to expire expiration timer value has changed. In an embodiment, the matching of the current value of the timer138to the next to expire expiration timer value may or may not indicate a failure of the master BMC. Therefore, if these values match the service processor130needs to take further action to determine whether the master BMC has failed, as will be described in greater detail below.

Thus, the service processor130(operating as the secondary BMC) determines whether the service processor120(operating as the master BMC) or one of service processors140and150(operating as slave BMCs) has failed. The service processor130can start this determination process by sending an alive message request504to the service processor120to determine whether the service processor120, operating as the master BMC, has failed by whether a primary alive message is received from service processor120in response to the alive message request504. The service processor130can also start a context switch process. In an embodiment, the context switch process can cause the service processor130to take over operating as the master BMC. In an embodiment, the start of the context switch process can include retrieving and loading the code associated with the master BMC functionality126into the processor memory112. In an embodiment, the master BMC functionality can include a reception address for the alive messages from service processors140and150, ownership of the storage location168in the shared system memory118, and the like.

After the service processor130has both sent the alive message request504and started the context switch process, the service processor130can set the timer138to a value equal to a time stamp plus an alive message period. In an embodiment, the alive message period can be a fixed period of time that the service processor120(operating as the master BMC) has to respond to the alive message request504if the service processor120has not failed. If the service processor130receives a primary alive message506from the service processor120within the alive message period, the service processor130can determine that the service processor120has not failed but that service processor140or150has failed. The service processor130can then stop the context switch process, can set the timer138to the next to expire expiration timer value from the storage location168, and can continue to operate as the secondary BMC. In this situation, the service processor120will have already updated expiration value corresponding to in the storage location168prior to the primary alive message506being sent to the service processor130.

However, if the alive message period ends and the timer138expires, the service processor130can determine that service processor120(operating as the master BMC) has failed. The service processor130can then complete the context switch process and begin operating as the master BMC. In an embodiment, when the service processor130begins operating as the master BMC, the service processor130can monitor the communication bus170for alive messages from service processors140and150, can have ownership to write data to the storage location168in the shared system memory118, and the like. The service processor130can then set the timer138to the next to expire expiration timer value in the storage location168, and send a rest signal508to the service processor120. In an embodiment, the service processor130can convert all of the expiration timer values in the storage location168to the clock of device104, such that further conversion of these values is not needed while the service processor130is the master BMC.

Upon the service processor120completing the reset, the service processor120can take over as the master BMC, and can provide a secondary signal to the service processor130. In an embodiment, the secondary signal can cause the service processor130to perform a context switch to begin operating as the secondary BMC. In an embodiment, the service processor120can convert all of the expiration timer values in the storage location168back to the clock of device102. In an embodiment, the reset operation of the service processor120can be a short enough time that a new secondary BMC is not needed. However, in another embodiment, the service processor130may determine that a new secondary BMC should be configured while service processor120is reset. In this embodiment, the service processor130can select either service processor140or service processor150to operate as the new secondary BMC while the service processor120is being reset. Thus, the distributed BMC functionality across the service processors120,130,140, and150can enable the server board100to continue to operate even if one of the BMCs fail.

FIG. 6illustrates a flow diagram of a method600for operating a service processor as a master baseboard management controller in accordance with at least one embodiment of the present disclosure. At block602, a first service processor is operated as a master baseboard management controller of a server board. In an embodiment, the first service processor can be the service processor120ofFIGS. 1-5. A second service processor is operated as a secondary baseboard management controller at block604. In an embodiment, the second service processor can be the service processor130ofFIGS. 1-5. At block606, a communication channel is monitored, by the first service processor, for alive messages from a plurality of service processors that includes the second service processor. In an embodiment, the service processors can communicate in compliance with the IPMI standard. At block608, a determination is made whether an alive message is received.

If an alive message is not received, a determination is made whether a timer has expired at block610. In an embodiment, the timer can a timer of the first service processor that is acting as the master baseboard management controller, and is set to a next to expire expiration timer value in a storage location of a shared memory in the server board. However, if an alive message is received at block608, the flow continues at block612and a value in a storage location is updated. In an embodiment, the value in the watch dog timer can correspond to a slave BMC service processor that provided the alive message to the first service processor. A primary expiration timer value is updated in the storage location at block614. In an embodiment, the primary expiration timer value can be the expiration timer value for the first service processor in the storage location. At block616, the timer is set to a value of a next to expire expiration value of the storage location.

Referring now to block610, if the timer has expired, the flow continues at block618and a reset signal is sent to one of the plurality of services processors. In an embodiment, the reset signal can be sent to the service processor corresponding to a current first to expire expiration value in the storage location. At block620, the current first to expire expiration value is updated in the storage location. In an embodiment, the current first to expire expiration value can be the expiration value for the service processor that is reset via the reset signal. In an embodiment, the updated expiration value is set equal to a current time stamp plus the amount of time needed to re-boot the service processor. The timer is set to a value of the next to expire expiration timer value at block622. In an embodiment, the next to expire expiration timer value is minimum value in the storage location.

FIG. 7illustrates a flow diagram of a method700for operating a service processor as a secondary baseboard management controller in accordance with at least one embodiment of the present disclosure. At block702, a first service processor is operated as a master baseboard management controller. In an embodiment, the first service processor can be the service processor120ofFIGS. 1-5. A second service processor is operated as a secondary baseboard management controller at block704. In an embodiment, the second service processor can be the service processor130ofFIGS. 1-5. At block706, a communication channel is monitored, by the second service processor, for a primary alive message from the first service processor. In an embodiment, the service processors can communicate in compliance with the IPMI standard. At block708, a determination is made whether a primary alive message is received.

If the primary alive message is received, a timer is set to a value of a next to expire expiration value in a storage location at block710. In an embodiment, the timer is located within the second service processor. If the primary alive message is not received, a determination is made whether the timer has expired at block712. If the timer has expired the flow continues at block714, otherwise the flow returns to block708. Thus, the combination of blocks708and712cause the second service processor to determine whether the primary alive message is received prior to the timer having expired. In an embodiment, the service processor130ofFIGS. 1-5can determine whether the primary alive message is received prior to the timer having expired.

At block714, a determination is made whether the value of the timer matches the next to expire expiration value in a storage location. If the values match the flow continues at block716below, otherwise the value of the timer is set to the next to expire expiration value in a storage location at block718and the flow continues as stated above at block706. At block716, a context switch process is started at the second service processor, and a primary alive message request is sent to the first service processor. In an embodiment, the start of the context switch process can include retrieving and loading the code associated with the master BMC functionality in a memory associated with the second service processor. In an embodiment, the master BMC functionality can include a reception address for the alive messages from slave BMC service processors, ownership of a storage location in a shared system memory, and the like. In an embodiment, the primary alive message request can be a request to determine whether the primary BMC is alive. The timer is set to an alive message period at block720. In an embodiment, the alive message period is a preset amount of time that the first service processor has to provide a primary alive message to the second service processor.

At block722, a determination is made whether the timer has expired. At block724, a determination is made whether the primary alive message is received. Blocks722and724form a loop, such that the second service processor can determine whether the primary alive message is received or the timer has expired first. In response to the primary alive message being received before the timer has expired, the flow continues at block726and the context switch process is ended. In response to the timer expiring before the primary alive message being received, the flow continues at block728and the first service processor is reset. At block730, the second service processor is operated as the master baseboard management controller.

According to one aspect, a server board is disclosed in accordance with at least one embodiment of the present disclosure. The server board includes a first device and a second device. The first device includes one or more general purpose processors configured to execute instructions. The first device also includes a first timer including a first storage location configured to store an expiration timer value. The first device further includes a first service processor that is configured as a master baseboard management controller of the server board to monitor a communication channel for alive messages from a plurality service processors corresponding to a plurality of other devices including the second device.

The second device includes one or more general purpose processors configured to execute instructions. The second device also includes a second timer including a second storage location configured to store an expiration timer value. The second device further includes a second service processor. The second service processor is configured as a secondary baseboard management controller of the server board when the first service processor is configured as the master baseboard management controller. The second service processor is configured to start a switchover process to operate as the master baseboard management controller, instead of the secondary baseboard management controller, in response to the second timer expiring based on a first value, and is configured to also reset the second timer to a second value based on an alive message period in response to the second timer expiring based on the first value. The second service processor is further configured to reset the first service processor and to complete the switchover process to operate as the master baseboard management controller in response to the second timer expiring based on the second value prior to a primary alive message being received from the first service processor.

In an embodiment, the first service processor is further configured to update a third expiration timer value of a third device in a storage location of a shared memory in response to an alive message being received from the third device, otherwise the first service processor is configured to send, to the third device, a reset signal in response to the first timer expiring prior to the alive message being received from the third device; the first service processor further configured to utilize expiration timer values in the storage location of the shared memory to update the first memory of the first timer. In an embodiment, the first memory of the first timer is updated with an expiration timer value of the shared memory in response to the alive message being received from the third service processor.

In an embodiment, the second service processor is further configured to reset the second timer to a next to expire expiration timer value in the storage location of the shared memory in response to receiving the primary alive message. In an embodiment, second service processor is further configured to, during the switchover process, send an alive message request to the first service processor, and to start a context switch to take over master baseboard management controller tasks.

In an embodiment, the first service processor is further configured to, in response to the first timer expiring, update a third expiration timer value, in the storage location of the shared memory, of a third device to create an updated third expiration timer value that equals a time stamp value plus a boot period value, to reset the third device, and reset the first timer to a value of a next to expire expiration value in the storage location of the shared memory. In an embodiment, the plurality of service processors operate as slave baseboard management controllers.

According to another aspect, a server board is disclosed in accordance with at least one embodiment of the present disclosure. The server board includes a first device, a second device, and a third device. The first device includes one or more general purpose processors to execute instructions. The first device also includes a first timer including a first storage location to store an expiration timer value. The first device further includes a first service processor. The first service processor is configured as a master baseboard management controller of the server board to monitor a communication channel for alive messages from a plurality service processors corresponding to a plurality of other devices including the second device and the third device. The first service processor is configured to update a third expiration timer value of the third device in a storage location of a shared memory in response to an alive message being received from the third device. The first service processor is further configured to send, to the third device, a reset signal in response to the first timer expiring prior to the alive message being received from the third device.

The second device includes one or more general purpose processors to execute instructions. The second device also includes a second storage location to store an expiration timer value. The second device further includes a second service processor. The second service processor is configured as a secondary baseboard management controller of the server board when the first service processor is configured as the master baseboard management controller. The second service processor is configured to start a switchover process to operate as the master baseboard management controller, instead of the secondary baseboard management controller, in response to the second timer expiring based on a first value, and is configured to also reset the second timer to a second value based on an alive message period in response to the second timer expiring based on the first value. The second service processor is further configured to reset first service processor and to complete the switchover process to operate as the master baseboard management controller in response to the second timer expiring based on the second value prior to a primary alive message being received from the first service processor.

In an embodiment, the first service processor is further configured to update an expiration timer value associated with the first service processor in the storage location of the shared memory in response to the alive message being received from the one of the third service processor. In an embodiment, the second service processor is further configured to reset the second timer to a next to expire expiration timer value in the storage location of the shared memory in response to receiving the primary alive message. In an embodiment, the second service processor is further configured to, during the switchover process, send an alive message request to the first service processor, and to start a context switch to take over master baseboard management controller tasks. In an embodiment, the first service processor is further configured to, in response to the first timer expiring, update a third expiration timer value of the third device to create an updated third expiration timer value that equals a time stamp value plus a boot period value, to reset a third service processor of the third device, and set the first timer to a value of a next to expire expiration value in the storage location of the shared memory. In an embodiment, the plurality of service processors operate as slave baseboard management controllers.

According to another aspect, a method is disclosed in accordance with at least one embodiment of the present disclosure. The method includes monitoring, by a first service processor, a communication channel for alive messages from a plurality of service processors corresponding to a plurality of other devices including a second device and a third device. In an embodiment, the plurality of service processors operate as slave baseboard management controllers. The method also includes updating, by the first service processor, a third expiration timer value of the third device in a storage location of a shared memory in response to an alive message being received from a third service processor of the third device over the communication channel. The method further includes sending, by the first service processor, a primary alive message to a second service processor of the second device in response to a first alive message being received from the third service processor. The method also includes resetting, by the first service processor, the third device in response to a first timer expiring prior to a first alive message being received from the third service processor.

In an embodiment, the method also includes operating, the second service processor of the second device, as a secondary baseboard management controller of the server board. The method further includes in response to a determination that a second timer has expired based on a first value: starting a switchover process; resetting the second timer to a second value based on an alive message period; and in response to a primary alive message not being received from the first service processor prior to the second timer expiring based on the second value: resetting the first service processor; and operating, the second service processor, as the master baseboard management controller.

In an embodiment, the method also includes setting, by the second service processor, the second timer to a next to expire expiration value in the storage location in response to receiving the primary alive message. In an embodiment, during the switchover process, the method further includes: sending, by the second service processor, an alive message request to the first service processor; and starting, by the second service processor, a context switch to take over master baseboard management controller tasks.

In an embodiment, the method further includes setting, by the first service processor, a first timer of the first service processor to a value of a next to expire expiration value in the storage location of the shared memory in response to the first alive message being received from the third service processor. In an embodiment, the method also includes updating, by the first service processor, a primary expiration value in the storage location of the shared memory in response to receiving the alive message from the third service processor. In an embodiment, the method further includes in response to the first timer expiring: updating, by the first service processor, a third expiration timer value of the third device to create an updated third expiration timer value that equals a time stamp value plus a boot period value; resetting a third service processor of the third device; and setting the first timer to a value of a next to expire expiration value in the storage location of the shared memory.

Other embodiments, uses, and advantages of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. The specification and drawings should be considered as examples only, and the scope of the disclosure is accordingly intended to be limited only by the following claims and equivalents thereof. For example, one skilled in the art would appreciate that a data processing system, such as a computer having an instruction based data processor, can be used to implement the analysis described herein.

Note that not all of the activities or elements described above in the general description are required, that a portion of a specific activity or device may not be required, and that one or more further activities may be performed, or elements included, in addition to those described. Still further, the order in which activities are listed are not necessarily the order in which they are performed.