Patent Description:
During pre-boot processes, the platform is initialized to specify the error propagation scheme (e.g., firmware first or O/S first) for each installed hardware device. Such hardware devices may include memory modules and Peripheral Component Interconnect express (PCIe) devices for accelerated networking, compute offloads, storage or other services. The error propagation scheme specified for a device cannot be changed during runtime.

A flexible error handling mechanism is desired by which an O/S may determine whether to handle errors in a suitable manner or provide control back to BIOS for typical firmware-first error handling. On-demand updates to an error propagation scheme for a particular hardware endpoint are also desirable, particularly updates which do not require hardware re-initialization.

<CIT> relates to OS and firmware coordinated error handling using transparent firmware intercept and firmware services. The system stack includes the OS error handling components, firmware, a platform, and a processor. The OS error handling components include a non-maskable interrupt (NMI) handler, a machine check exception handler, and a corrected error handler. The firmware includes a CSF interface, a system management interrupt (SMI) firmware block, and firmware error handlers. The Sensor Effector Interface (SEI) also provides a framework such that modular and device independent code can be interpreted and run within this framework. The SMI causes the platform processor to switch to its system management mode (SMM), a special operating mode that is hidden from the OS. In response to an SMI, a set of SMM handlers (stored in a partitioned portion of system memory called SMRAM) is walked to identify an appropriate handler, which is then used to perform firmware preprocessing operations. Upon completion of these operations at time T2, an SMM return instruction is executed to cause the processor to return to its previous execution context, resuming execution of the operating system's current thread. At substantially the same time, the processor or platform informs the OS of the hardware error event. The OS then performs OS error handling until time T3, at which point the OS may submit a request for firmware error-handling services via CSF interface.

<CIT> relates to a system and method for using hot plug configuration for PCI error recovery. Errors resulting from failures in devices coupled to a bus or interconnect having hot plug configuration capabilities such as a peripheral component interconnect (PCI) bus are handled. Advanced configuration and power interface (ACPI) hot plug capabilities, such as those used in PCI-X bus architectures, are leveraged together with PCI error detection functionality to provide an error recovery mechanism that enables a faulty bus device to be taken offline without having to reboot the system. The error handling method combines the error detection and notification functions performed by PCI error reporting module with the error handling routines and instructions performed by a PCI error handler deployed from ACPI BIOS.

It is the object of the present invention to improve error handling on a server platform to reduce downtime.

The following description is provided to enable any person in the art to make and use the described embodiments. Various modifications, however, will remain readily-apparent to those in the art.

Some embodiments provide a technical solution to the technical problem of fixed error propagation schemes by determining at the O/S whether to handle an error in the O/S or the firmware based on updatable filtering logic. Embodiments may also provide a technical solution to the problem of excessive downtime during platform customization/update by allowing updates to the filtering logic without requiring hardware re-initialization.

Generally, some embodiments operate to detect a hardware endpoint error and trigger a software interrupt received by an Advanced Configuration and Power Interface (ACPI) runtime component. The ACPI runtime component determines whether the error is designated to be handled by a firmware-based error handler and, if so, trigger a software interrupt to a firmware component for handling the error. If the error is not designated to be handled by a firmware-based error handler, the error is handled by a suitable O/S error handler.

The determination of whether the error is designated to be handled by a firmware-based error handler may be based on the presence or absence of a corresponding ACPI device-specific method in the O/S layer. Advantageously, and according to some embodiments, the error propagation scheme for a hardware endpoint may be changed without changing the platform BIOS, but simply by updating one or more ACPI device-specific methods and O/S handlers. In some instances, the ACPI device-specific methods and O/S handlers may be updated (and the error propagation scheme thereby changed) without re-initializing the platform, thereby minimizing system downtime.

<FIG> is a block diagram of platform architecture <NUM> according to some embodiments. Platform architecture <NUM> may represent a server runtime architecture. Architecture <NUM> may support user-mode applications and/or virtual machines as is known in the art.

Hardware layer <NUM> of architecture <NUM> includes Central Processing Unit (CPU) <NUM>, which is configured to detect errors from various hardware endpoints <NUM>-<NUM>. According to some embodiments, a hardware endpoint <NUM>-<NUM> errors are detected by a controller or port of CPU <NUM>. Embodiments may include more than one CPU, and each of such CPUs may receive error notifications from any number of hardware endpoints.

Hardware layer <NUM> also includes Service Processor <NUM>. Service Processor <NUM> may comprise a microcontroller distinct from CPU <NUM> and which manages platform-specific functions such as monitoring environmental conditions and handling certain error conditions. Service Processor <NUM> may persist error records within a system event log for post-crash analysis, for example, and may comprise a component of a Baseboard Management Controller.

BIOS layer <NUM> includes runtime components which are initialized during BIOS boot-up. Specifically, System Management Mode (SMM) runtime <NUM> is initialized and associated with device-specific error handlers <NUM>. ACPI runtime <NUM> is also initialized during BIOS boot.

BIOS boot may further include configuration of error signaling as an ACPI System Control Interrupt (SCI) as opposed to an SMI (System Management Interrupt) propagated to SMM runtime <NUM>. This configuration may be specified via a General-Purpose IO (GPIO) setting of a Platform Controller Hub (not shown) as is known in the art.

O/S layer <NUM> includes runtime components instantiated during booting of the O/S following the BIOS boot. Error-handling filters <NUM> may comprise Device-Specific Methods (_DSM) associated with one or more hardware endpoints. Usage of error-handling filters <NUM> will be described in detail below.

Error handlers <NUM>, kernel <NUM> and O/S event log <NUM> of O/S layer <NUM> may operate as is known in the art. For example, an error handler <NUM> corresponding to the hardware having the error condition creates an error packet and forwards the packet to kernel <NUM>. Kernel <NUM> creates an error record based on the error packet and may log the error record among error records <NUM> of O/S event log <NUM>. In the case of a nonfatal uncorrected hardware error, kernel <NUM> may attempt to correct the error condition.

<FIG> is a block diagram of platform <NUM> according to some embodiments. Platform <NUM> may implement architecture <NUM>, but embodiments are not limited thereto.

Platform <NUM> comprises two CPUs <NUM> and <NUM>, which may comprise any types of CPUs that are or become known. Embodiments are not limited to platforms including any particular number of CPUs. Each CPU <NUM> and <NUM> is connected to various I/O interfaces into which corresponding hardware I/O endpoints may be installed.

For example, CPU <NUM> is coupled to Dual In-line Memory Modules (DIMMs) <NUM> and <NUM>. CPU <NUM> includes integrated memory controller <NUM> to interface with DIMMs <NUM>. Accordingly, memory controller <NUM> operates to detect errors in DIMMs <NUM> and <NUM> and to take appropriate corrective actions, if possible. Memory controller <NUM> also logs error information into architected error context registers, e.g., Memory-Specific Registers (MSRs) of CPU <NUM>. CPU <NUM> is coupled directly to DIMMs <NUM> and <NUM> and includes memory controller <NUM> which may operate as described above.

CPU <NUM> is also directly coupled to PCIe slots <NUM>, <NUM> and <NUM>. PCIe slots <NUM>, <NUM> and <NUM> may mount expansion cards providing networking, graphics processing, storage or other functionality. Each expansion card may communicate directly with a PCIe root port of CPU <NUM> through its associated PCIe slot <NUM>, <NUM> or <NUM> and the bus connecting the slot with CPU <NUM>. The PCIe root port of CPU <NUM> may detect errors on ay PCIe endpoints to which it is connected. Again, CPU <NUM> is also coupled to PCIe slots <NUM> and <NUM> and may include a PCIe root port to detect errors from PCIe endpoints installed therein.

CPU <NUM> is coupled to Platform Controller Hub (PCH) <NUM> as is known in the art. PCH <NUM> controls certain data paths and support functions used in conjunction with CPUs <NUM> and <NUM>. PCH <NUM> is coupled to Flash memory <NUM> storing the BIOS as well as to Baseboard Management Controller (BMC) <NUM>. BMC <NUM> is a specialized service processor that monitors the physical state of platform <NUM>, and may support out-of-band communications with a system administrator through management interface <NUM>.

<FIG> comprises a flow diagram of process <NUM> according to some embodiments. In some embodiments, processing units (e.g., one or more processors, processing cores, processor threads) of a computing device (e.g., a computer server) execute program code to cause the device to perform process <NUM>. Process <NUM> and all other processes mentioned herein may be embodied in processor-executable program code read from one or more of non-transitory computer-readable media, such as a hard disk, a Flash drive, etc., and then stored in a compressed, uncompiled and/or encrypted format. In some embodiments, hard-wired circuitry may be used in place of, or in combination with, program code for implementation of processes according to some embodiments. Embodiments are therefore not limited to any specific combination of hardware and software.

Process <NUM> commences upon detection of a memory error at S310. As described above, a platform memory controller (which may or may not be integrated into a platform CPU) may detect a memory error at S310 using known error signaling and detection protocols. The memory controller may take corrective actions and may also log corresponding error information into architected error context registers of a platform CPU.

The logged error information causes the CPU to trigger an interrupt at S320. The interrupt may comprise an SCI which is received by an ACPI runtime component as illustrated in <FIG>. Next, at S330, the ACPI runtime component determines whether the error should be handled by a firmware-based error handler in the BIOS runtime, or by an O/S error handler.

The determination at S330 may include determining whether the O/S includes a device-specific method associated with the error source. If no such method exists, flow proceeds to S340 to handle the interrupt using an O/S error handler (e.g., handler <NUM>). Advantageously, such error handling may proceed on the processing thread which receives the error, leaving other processing threads to execute O/S functions.

If a device-specific method exists, the interrupt is handled at S350 by an error handler registered with the BIOS runtime (e.g., with the System Management Mode runtime). In this regard, the ACPI runtime may raise a System Management Interrupt (SMI) as shown in <FIG> to trigger the SMM error handler. Flow then continues from S350 to S340 to pass control to the O/S error handler.

Process <NUM> need not be limited to memory errors, but may be extended to handle errors detected for any hardware endpoint. Particular implementations of error detection and raising of the SCI may differ depending upon the hardware endpoint. Process <NUM> of <FIG> depicts processing of detected PCIe errors according to some embodiments.

A PCIe error is detected at S410. Each PCIe endpoint installed in a platform is associated with a PCIe root port. A PCIe root port may detect errors from its associated PCIe device. PCIe root ports may be integrated into a CPU, a PCH or any suitable hardware component.

In response to detection of an error, the PCIe root port triggers a PCIe Advanced Error Reporting (AER) Message Signal Interrupt (MSI) at S420. The MSI is received by an O/S root port error handler, which triggers an SCI in response thereto. As described above, the SCI is received by an ACPI runtime component, which determines at S430 whether the error should be handled by a firmware-based error handler in the BIOS runtime.

The determination at S430 may proceed similarly to that described above with respect to S330. Specifically, if a device-specific method associated with the error source exists, the ACPI runtime may raise an SMI to trigger a corresponding SMM error handler to handle the error at S440. Flow then proceeds to S450 to return control to the O/S root port error handler. If no corresponding device-specific method is identified at S430, the error is returned to the O/S root port error handler at S450 without first returning control to the BIOS.

Process <NUM> of <FIG> illustrates an alternative approach to handling PCIe errors according to some embodiments. S510, S520, S530 and S540 may proceed similarly to S410, S420, S430 and S450 as described above in a case that it is determined that a detected PCIe error is to be handled exclusively by an O/S root port handler.

However, if it is determined at S530 that a detected PCIe error is to be handled by an SMM handler, flow initially proceeds to S550 to handle the error using the SMM handler. After the SMM handler completes its processing, it is determined whether the error is a correctable error at S560. If so, an SCI is triggered at S570 to return control to the ACPI runtime. If the error is determined to be uncorrectable at S560, a Non-Maskable Interrupt (NMI) is triggered at S580. An NMI is reported to the O/S regardless of the processor's current interrupt priority level. An NMI usually indicates a fatal hardware error condition and is acted on by the O/S accordingly.

<FIG> illustrates replacement of a PCIe endpoint installed in socket <NUM> of platform <NUM> with a non-identical (e.g., upgraded) PCIe endpoint. Replacement of such an endpoint may conventionally require a firmware update or at least a system reboot to ensure proper error handling.

<FIG> illustrates process <NUM> for responding to a replaced hardware endpoint according to some embodiments. At S710, an endpoint is replaced during O/S execution. For example, during the replacement of an endpoint within a platform (e.g., a computer server) at S710, the platform has fully booted into its O/S and the O/S is running to serve requests received from applications.

Next, at S720, the O/S is updated to disassociate the endpoint with a BIOS error handler. It is therefore assumed that the prior endpoint was associated with an SMM error handler and its errors were first handled by the SMM error handler as described with respect to processes <NUM> and <NUM>. As also described above, some embodiments may associate an error source with an SMM error handler using an ACPI device-specific method. Accordingly, S720 may comprise disabling or uninstalling an ACPI device-specific method which associates the prior endpoint with an SMM error handler. The O/S is then updated at S730 to include an O/S error handler associated with the replacement endpoint.

At S740, the platform is rebooted to the now-updated O/S without executing the BIOS. In some embodiments, S740 consists of a Kernel Soft Reboot (KSR) which reboots the O/S without re-initializing the platform hardware. For example, control is handed to a driver rather than to firmware after shutdown, and proceeds directly to kernel initialization while bypassing platform initialization, the boot manager and the O/S loader. A KSR may advantageously allow execution of an updated O/S while experiencing minimal platform downtime.

By virtue of the foregoing, and according to processes <NUM> and <NUM>, detected errors from the new endpoint will be handled by the newly-included O/S handler, rather than by an SMM handler associated with the prior endpoint. Such an arrangement allows introduction of new endpoint hardware error handling into the platform without requiring a firmware update. Moreover, using a KSR or other similar technology, the platform downtime is significantly reduced.

Each functional component described herein may be implemented in computer hardware (integrated and/or discrete circuit components), in program code and/or in one or more computing systems executing such program code as is known in the art. Such a computing system may include one or more processing units which execute processor-executable program code stored in a memory system.

Claim 1:
A computing system comprising:
an I/O hardware endpoint;
an I/O hardware endpoint controller in communication with the I/O hardware endpoint and to:
detect (S310; S410) an error on the I/O hardware endpoint; and
trigger (S320, S420) an interrupt in response to detected error; and
a processor to execute an operating system runtime component to:
receive the interrupt at the operating system runtime component;
in response to the received interrupt, determine (S330; S430) at the operating system runtime component whether to handle the error using an operating system handler or a firmware runtime error handler associated with the I/O hardware endpoint; and
if it is determined to handle the error using the firmware runtime error handler associated with the I/O hardware endpoint, trigger (S350; S440) a firmware interrupt associated with the firmware runtime error handler;
wherein the processor, while executing the operating system runtime component, is further configured to, in response to the I/O hardware endpoint being replaced:
disassociate (S720) the I/O hardware endpoint with the firmware runtime error handler;
update (S730) the operating system to include an operating system error handler associated with the replacement I/O hardware endpoint; and
reboot (S740) the operating system without re-initializing the hardware of the computing system.