Software handling of hardware error handling in hypervisor-based systems

Errors occurring on a hardware bus of a hypervisor-based system may be handled in software monitors in the hypervisor-based system. When an error occurs, guest partitions on the hypervisor-based system may be notified of the error through a monitor executing in each guest partition. Only guest partitions affected by the error may be shut down or provided other instructions for taking an action in response to the error.

The instant disclosure relates to error handling. More specifically, this disclosure relates to error handling in hypervisor-based systems.

BACKGROUND

Hypervisor-based systems may execute multiple operating systems within through multiple guest partitions. The guest partitions share access to hardware in the hypervisor-based system. The hardware in the hypervisor-based system may be designed to provide robust error reporting. Conventionally, errors reported by the hardware are either correctable or uncorrectable. Correctable errors are treated as warnings. The uncorrectable errors are handled differently, based on whether they are non-fatal or fatal. Non-fatal errors are serious, but the system may handle these errors through other means, such as redundant paths. Fatal errors are errors that affect the integrity of operations in the system, and may cause serious system reliability issues if the system continues to operate without taking a corrective action.

When error reporting is enabled in a single partition system, a fatal uncorrectable error may deliberately cause the system to reboot or shut down to prevent further unintended damage, such as data corruption. However, a system shutdown is not a desirable approach in a hypervisor-based system having multiple partitions residing in a single system. When a system shutdown occurs, all guest partitions on the system become unavailable. Thus, an error in one guest partition results in unavailability of all guest partitions. Maintaining reliability of the hypervisor-based system may be difficult in these circumstances. For example, additional planning may be required to ensure critical software does not execute on guest partitions sharing hardware with unreliable software. Such planning may be difficult, because guest partitions generally cannot access other guest partitions in the hypervisor-based system.

SUMMARY

According to one embodiment, a method includes receiving, from a bus of a hypervisor system, an error interrupt. The method also includes identifying at least one affected guest partitions from a plurality of guest partitions executing on the hypervisor system. The method further includes broadcasting the error interrupt to the affected guest partitions. [This is a repeat of the claim language at the end of the document. You may skip this section in your review and review the claims at the end of the document.]

According to another embodiment, a computer program product includes a non-transitory computer readable medium having code to receive, from a bus of a hypervisor system, an error interrupt. The medium also includes code to identify at least one affected guest partitions from a plurality of guest partitions executing on the hypervisor system. The medium further includes code to broadcast the error interrupt to the affected guest partitions.

According to a further embodiment, an apparatus includes a bus and a processor coupled to the bus. The processor is configured to receive, from a bus of a hypervisor system, an error interrupt. The processor is also configured to identify at least one affected guest partitions from a plurality of guest partitions executing on the hypervisor system. The processor is further configured to broadcast the error interrupt to the affected guest partitions.

DETAILED DESCRIPTION

An error generated by one device may be contained within the partitions that share the device. This reduces the likelihood that an error on the hardware bus creates an interruption in the normal operation of the other guest partitions. The error may be handled in software as the hypervisor-based system having the device that caused the error.

FIG. 1is a flow chart illustrating a method of handling bus errors in software according to one embodiment of the disclosure. A method100begins at block102with a hypervisor-based system receiving an error interrupt on a hardware bus. At block104, the affected guest partitions are identified. The affected guest partitions may be determined by identifying guests partitions owning the device that generated the error interrupt. At block106, the error interrupt is broadcast to the affected guest partitions through a software monitor. A monitor executing on each guest partition may receive the error interrupt and take appropriate corrective action. For example when the error in uncorrectable, the affected guest partition may shut down. In another example when the error is correctable, the affected guest partition may repeat a read or write request to a memory device coupled to the hardware bus.

According to one embodiment, the error interrupt may be broadcast to all guest partitions. A monitor installed in each of the guest partitions may receive the broadcast error interrupt and determine whether the source of the error interrupt is a device assigned to the guest partition. If the error interrupt does not affect the guest partition, the guest partition may continue operating normally. If the error interrupt does affect the guest partition, the guest partition may wait for a message from a central error handling routine on a service partition, such as an advanced configuration and power interface (ACPI) service partition, of the system executing the guest partitions. The service partition may identify affected guest partitions assigned to the device that generated the error interrupt and transmit messages to the affected guest partitions. The message may include a corrective action for the guest partition to take, such as rebooting the guest partition. Alternatively, the message may include an instruction to remove and clear the device that caused the error.

According to one embodiment, the hardware bus may be a peripheral component interface express (PCI-e) bus. Advanced error reporting (AER) in PCI-e employs hardware that resides in hierarchical components including root ports, switches, and devices. Root ports, switches, and devices may have AER enabled or disabled. Software may be programmed to access and control the AER-enabled components on the PCI-e bus. In a virtualized environment, a guest partition or virtual machine may only see a device or particular exposed function of a device. The hypervisor-based system executing the guest partitions may interface with devices to handle error reporting and report errors to the guest partitions affected by the error that otherwise would not receive the error from the hardware bus due to limited access to the hardware devices.

FIG. 2is block diagram illustrating components coupled to a bus involved at various aspects of error reporting according to one embodiment of the disclosure. A system200includes a root port204. The root port204may have memory storage, which includes correctable, non-fatal, and fatal error registers. Other root port registers may include a root error command register, a root error status register, an error source identification register, a root control register, a device control register, and a bridge control register. The root error status register and error source identification registers may store information regarding an error after an error interrupt occurs. A switch206coupled to the root port204may include memory, such as correctable, non-fatal, and fatal error registers. Other registers in the switch206may include a device control register and a bridge control register. Devices208and210may be coupled to the root port204, through the switch206. A device212may also be coupled directly to the root port204. Devices208,210, and212may include memory storage such as correctable, non-fatal, and fatal error registers. Other registers in the devices208,210, and212may include a device control register.

When an error occurs in one of the devices208-212or the switch206, error bits may be set in their error status registers. Then, a header log and a message may be sent to the root port204. The root port204may modify an error source identification register based, in part, on the message and generate an error interrupt transmitted to hardware and/or software is coupled to the root port204, such as a hypervisor-based system. Alternatively, when the root port204detects an error, the root port204may set status bits and an error source identification before generating the interrupt.

FIG. 3is logic diagram illustrate a logic flow for error message controls in a peripheral component interface express (PCI-e) system according to one embodiment of the disclosure. A logic diagram300includes logic gates representing operations performed during error handling. An active path302-312through the logic diagram300illustrates paths for handling errors occurring on the hardware bus according to one embodiment of the disclosure. First, an error message may be received at path302. The error message may be transmitted through path304a,306,308, and310to generate an error interrupt at path312. Alternatively, an internally generated error message may propagate from path304bthrough paths306,308, and310to generate an error interrupt at path312. The error interrupt generated in path312may be transmitted as a system notification to the hypervisor-based system, which then identifies affected guest partitions and alerts the affected guest partitions.

During propagation of the error through the logic flow300ofFIG. 3, devices, such as the devices208-212ofFIG. 2, may modify bits corresponding to correctable error reporting, non-fatal error reporting, fatal error reporting, and unsupported request reporting in the control register of the respective device. Additionally, a parity error response bit and a SERR# bit may be set in the command register of the device. In parallel, switches, such as the switch206ofFIG. 2, may enable correctable error reporting bits, non-fatal error reporting bits, fatal error reporting bits, and/or unsupported request reporting bits in the control register of the switch. Additionally, set parity error response bits may be cleared in the command register, and the system error (SERR#) bit set in the bridge control register. The root port, such a the root port204ofFIG. 2, may enable a correctable error reporting bit, a non-fatal error reporting bit, a fatal error reporting bit, and an unsupported request reporting bit in the device control register. Additionally, a parity error response bit may be set, a SERR# bit may be cleared in the command register, and the SERR# bit may be set in the bridge control register. The root port may further ensure system error on correctable, non-fatal, and fatal error enables are off in the root control register, and set the correctable, non-fatal, and fatal error reporting bits in the root error command register.

After the error interrupt is generated and distributed to affected guest partitions, a software monitor function in each guest partition may take corrective action. According to one embodiment, the error interrupt may be broadcast to all instances of monitor software in the guest partitions. Decisions on what action the guest takes in response may be specific to how the error affects that guest, if at all. Thus, the effect on guest partitions of an error interrupt in other guest partitions is reduced, because not all guest partitions are shut down when an error only affects some of the guest partitions.

Certain errors reported through AER in the PCI-e bus may be determined to be fatal and receive a certain response from the hypervisor-based system or the guest partitions when such an error is generated. Table 1 provides a list of error codes, the corresponding error, and whether such an error is fatal.

Correctable errors are listed in Table 2, including a threshold value for a number of allowed correctable errors of each type before an action is taken. For example, when individual correctable errors occur a first response may be taken. When a specified number of correctable errors of a certain type have occurred, a function may be called to notify the system and reset the counter for that particular correctable error.

FIG. 4is a flow chart illustrating a method of handling correctable errors in a peripheral component interface express (PCI-e) bus according to one embodiment of the disclosure. Correctable errors may be serviced by an AER Handling Module (AHM) executing in a service partition, such as an advanced configuration and power interface (ACPI) service partition, of the hypervisor-based system. According to one embodiment, the ACPI service partition is a self-contained partition that services AERs. The hardware may be set to not generate a message signaled interrupt (MSI) on this category of errors.

A method begins at block402with the AHM periodically polling the root port's correctable error status register for errors. At block404, it is determined if a bit in the register is set. If any of the bits in the correctable error registers are set, the AHM may collect and log information regarding the error at block406before returning to polling at block402. In one embodiment, AER may not route any AER messages to the affected guest partitions but handle the AER message locally. In another embodiment, in addition to handling AERs locally, AERs may also be routed to affected guests partitions to take advantage of the AER handling mechanism in the operating system of each guest partition.

FIG. 5is a flow chart illustrating a method of handling uncorrectable errors in a peripheral component interface express (PCI-e) bus according to one embodiment of the disclosure. A method500for handling an uncorrectable error begins at block502with receiving an uncorrectable error interrupt at a logical CPU (lCPU). At block504, the uncorrectable error bit for each virtual CPU (vCPU) of the lCPU is set. At block506, it is determined if the uncorrectable error affects any of the vCPUS of the lCPU. If none of the vCPUs are affected by the uncorrectable error, the method500returns to block502until another uncorrectable error for the lCPU is received. If the error does affect a vCPU, the method500continues to block508to signal the guest partition corresponding to the vCPU to take an action, such as shutting down. [The method ofFIG. 5is written from the “view” of the error handling module, rather than the guest partitions. Thus, although the guest partition may be monitoring for a service message we did not discuss it in this paragraph (although we did in other paragraphs). Please let us know if there is an inaccuracy or missing information.]

An uncorrectable error may be contained to affect only the guest partition or guest partitions that own the device or switch that generated the error interrupt. According to one embodiment, after an uncorrectable error occurs, guest partitions may be prevented from completing other outstanding commands with potentially undetected errors. For example, the guest partition may be prevented from receiving additional interrupts and/or other input/output (I/O) requests. Further, the affected guests may be prevented from servicing any interrupts until the error interrupt is handled. The other interrupts may be paused by setting an AER interrupt with a highest priority.

An AER message signaled interrupt (MSI) may be set to broadcast to all logical central processor units (lCPUs) in the hypervisor-based system, which prevents all the guest partitions from servicing any interrupts until the affected guest partitions are isolated. Monitors in each guest partition check for these broadcasts and may not be permitted to resume the guest partition until the check is complete. A bit, such as an AER Notification Flag (ANF), may be assigned to each of the vCPUs (virtual CPUs) of a lCPU (logical CPU). The ANF bit may be an indicator to a vCPUs monitor as to whether it should check for unrecoverable errors prior to resuming the guest partition. When an AER interrupt is received on an lCPU, the monitor servicing the interrupt on that lCPU may set all the ANF bits of all the vCPUs for that lCPU. Then, the monitor may check for uncorrectable errors. After checking for uncorrectable errors, the monitor clears the ANF bit for each vCPU and resumes the guest partition if no error bits are set by any devices owned by that guest partition. The same checks may be repeated each time a vCPU wakes up for a new quantum on their lCPU and the ANF bit for the vCPU is set.

When a monitor detects an uncorrectable error in any device owned by the guest partition, the guest partition may be paused and a state change message issued notifying the hypervisor-based system. After all vCPUs have woken up, all of the vCPUs' monitor instances will have serviced their ANF bits. The monitor instances of the affected vCPUs and the corresponding guest partition may then wait until an indication is received that provides instructions for the guest partitions.

According to one embodiment, the AER handling module (AHM) executing in the ACPI service partition may have access to error status registers of all the devices and switches in the hypervisor-based system. The AHM may signal to a monitor executing for a particular vCPU, corresponding to an affected guest partition, to force a system dump and reboot.

The AER Handling Module (AHM) in the ACPI service partition may be responsible for servicing both correctable and uncorrectable errors. In this embodiment, the AHM may periodically check a correctable error status register of all the root devices for correctable errors for one or more set bits, indicating a correctable error has occurred. Further, the AHM may respond to uncorrectable errors when an error interrupt occurs. Following the error interrupt, the AHM may read status registers under a root port and corresponding devices to discover a source of the error interrupt. Then, the AHM may log the error, send a message to the hypervisor-based system, force a system dump, and/or send a message to the guest partitions that are affected to shutdown.

FIG. 6illustrates one embodiment of a system600for an information system, such as a system including a hypervisor-based system. The system600may include a server602, a data storage device606, a network608, and a user interface device610. The server602may be a dedicated server or one server in a cloud computing system. The server602may also be a hypervisor-based system executing one or more guest partitions. In a further embodiment, the system600may include a storage controller604, or storage server configured to manage data communications between the data storage device606and the server602or other components in communication with the network608. In an alternative embodiment, the storage controller604may be coupled to the network608.

In one embodiment, the user interface device610is referred to broadly and is intended to encompass a suitable processor-based device such as a desktop computer, a laptop computer, a personal digital assistant (PDA) or tablet computer, a smartphone or other a mobile communication device having access to the network608. When the device610is a mobile device, sensors (not shown), such as a camera or accelerometer, may be embedded in the device610. When the device610is a desktop computer the sensors may be embedded in an attachment (not shown) to the device610. In a further embodiment, the user interface device610may access the Internet or other wide area or local area network to access a web application or web service hosted by the server602and provide a user interface for enabling a user to enter or receive information.

The network608may facilitate communications of data, such as authentication information, between the server602and the user interface device610. The network608may include any type of communications network including, but not limited to, a direct PC-to-PC connection, a local area network (LAN), a wide area network (WAN), a modem-to-modem connection, the Internet, a combination of the above, or any other communications network now known or later developed within the networking arts which permits two or more computers to communicate.

In one embodiment, the user interface device610accesses the server602through an intermediate sever (not shown). For example, in a cloud application the user interface device610may access an application server. The application server fulfills requests from the user interface device610by accessing a database management system (DBMS). In this embodiment, the user interface device610may be a computer or phone executing a Java application making requests to a JBOSS server executing on a Linux server, which fulfills the requests by accessing a relational database management system (RDMS) on a mainframe server.

FIG. 7illustrates a computer system700adapted according to certain embodiments of the server602and/or the user interface device610. The central processing unit (“CPU”)702is coupled to the system bus704. The CPU702may be a general purpose CPU or microprocessor, graphics processing unit (“GPU”), and/or microcontroller. The present embodiments are not restricted by the architecture of the CPU702so long as the CPU702, whether directly or indirectly, supports the operations as described herein. The CPU702may execute the various logical instructions according to the present embodiments.

The computer system700also may include random access memory (RAM)708, which may be synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), or the like. The computer system700may utilize RAM708to store the various data structures used by a software application. The computer system700may also include read only memory (ROM)706which may be PROM, EPROM, EEPROM, optical storage, or the like. The ROM may store configuration information for booting the computer system700. The RAM708and the ROM706hold user and system data.

The computer system700may also include an input/output (I/O) adapter710, a communications adapter714, a user interface adapter716, and a display adapter722. The I/O adapter710and/or the user interface adapter716may, in certain embodiments, enable a user to interact with the computer system700. In a further embodiment, the display adapter722may display a graphical user interface (GUI) associated with a software or web-based application on a display device724, such as a monitor or touch screen.

The I/O adapter710may couple one or more storage devices712, such as one or more of a hard drive, a solid state storage device, a flash drive, a compact disc (CD) drive, a floppy disk drive, and a tape drive, to the computer system700. According to one embodiment, the data storage712may be a separate server coupled to the computer system700through a network connection to the I/O adapter710. The communications adapter714may be adapted to couple the computer system700to the network608, which may be one or more of a LAN, WAN, and/or the Internet. The communications adapter714may also be adapted to couple the computer system700to other networks such as a global positioning system (GPS) or a Bluetooth network. The user interface adapter716couples user input devices, such as a keyboard720, a pointing device718, and/or a touch screen (not shown) to the computer system700. The keyboard720may be an on-screen keyboard displayed on a touch panel. Additional devices (not shown) such as a camera, microphone, video camera, accelerometer, compass, and or gyroscope may be coupled to the user interface adapter716. The display adapter722may be driven by the CPU702to control the display on the display device724. Any of the devices702-722may be physical, logical, or conceptual.

The applications of the present disclosure are not limited to the architecture of computer system700. Rather the computer system700is provided as an example of one type of computing device that may be adapted to perform the functions of a server602and/or the user interface device610. For example, any suitable processor-based device may be utilized including, without limitation, personal data assistants (PDAs), tablet computers, smartphones, computer game consoles, and multi-processor servers. Moreover, the systems and methods of the present disclosure may be implemented on application specific integrated circuits (ASIC), very large scale integrated (VLSI) circuits, or other circuitry. In fact, persons of ordinary skill in the art may utilize any number of suitable structures capable of executing logical operations according to the described embodiments. For example, the computer system700may be virtualized for access by multiple users and/or applications.

FIG. 8Ais a block diagram illustrating a server hosting an emulated software environment for virtualization according to one embodiment of the disclosure. An operating system802executing on a server includes drivers for accessing hardware components, such as a networking layer804for accessing the communications adapter614. The operating system802may be, for example, Linux. An emulated environment808in the operating system802executes a program810, such as CPCommOS. The program810accesses the networking layer804of the operating system802through a non-emulated interface806, such as XNIOP. The non-emulated interface806translates requests from the program810executing in the emulated environment808for the networking layer804of the operating system802.

In another example, hardware in a computer system may be virtualized through a hypervisor.FIG. 8Bis a block diagram illustrating a server hosing an emulated hardware environment according to one embodiment of the disclosure. Users852,854,856may access the hardware860through a hypervisor858. The hypervisor858may be integrated with the hardware860to provide virtualization of the hardware860without an operating system, such as in the configuration illustrated inFIG. 8A. The hypervisor858may provide access to the hardware860, including the CPU702and the communications adaptor714.