Patent Publication Number: US-6658599-B1

Title: Method for recovering from a machine check interrupt during runtime

Description:
BACKGROUND OF THE INVENTION 
     1. Technical Field 
     The present invention relates generally to the field of computer architecture and, more specifically, to methods and systems for managing machine check interrupts during runtime. 
     2. Description of Related Art 
     A logical partitioning option (LPAR) within a data processing system (platform) allows multiple copies of a single operating system (OS) or multiple heterogeneous operating systems to be simultaneously run on a single data processing system platform. A partition, within which an operating system image runs, is assigned a non-overlapping sub-set-of the platform&#39;s resources. These platform allocable resources include one or more architecturally distinct processors with their interrupt management area, regions of system memory, and input/output (I/O) adapter bus slots. The partition&#39;s resources are represented by its own open firmware device tree to the OS image. 
     Each distinct OS or image of an OS running within the platform is protected from each other such that software errors on one logical partition can not affect the correct operation of any of the other partitions. This is provided by allocating a disjoint set of platform resources to be directly managed by each OS image and by providing mechanisms for ensuring that the various images can not control any resources that have not been allocated to it. Furthermore, software errors in the control of an OS&#39;s allocated resources are prevented from affecting the resources of any other image. Thus, each image of the OS (or each different OS) directly controls a distinct set of allocable resources within the platform. 
     Currently, in both LPAR systems and non-partitioned systems, when a machine check occurs due to a bad I/O adapter in the system, data about the condition causing the machine check is presented to the operating system in the form of an error log entry. The operating system then performs a complete shutdown of the system. The user must then replace the bad I/O adapter and then reboot the system. Such a requirement may not be terribly problematic for users with a simple configuration in which a reboot is relatively quick or for users in which having the system available at all times is not critical. However, for other users with complex configurations, such as, for example, multiple racks of serial storage architecture (SSA) or networked systems, a considerable amount of time will be spent rebooting the system just to replace one bad I/O adapter. Such expenditure of time may be very costly for those users. For example, if the system is a web server critical for taking internet sales orders for products, such as, for example, books or compact disks (CDs), each minute of time that the system is shut down to replace a bad I/O adapter may result in many thousands of dollars in lost sales. 
     Therefore, a method and system for replacing bad I/O adapters without the need for powering down or rebooting the system would be desirable. 
     SUMMARY OF THE INVENTION 
     The present invention provides a method, system, and apparatus for managing a failed input/output adapter within a data processing system. In one embodiment, an operating system handler receives an indication that one of a plurality of input/output adapters has failed. The operating system handler consults an error log to determine which input/output adapter has failed. Once the bad input/output adapter has been determined, the operating system handler disables the bad input/output adapter and deallocates any processes bound for the bad input/output adapter without powering down the data processing system. A user is then notified of the bad input/output adapter so that the bad input/output adapter can be replaced. The input/output adapter may be replaced without powering down the data processing system. Once the bad input/output adapter has been replaced, the new input/output adapter is enabled. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The novel features believed characteristic of the invention are set forth in the appended claims. The invention itself, however, as well as a preferred mode of use, further objectives and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein: 
     FIG. 1 depicts a block diagram of a data processing system in accordance with the present invention; 
     FIG. 2 depicts a block diagram of a system for handling machine check interrupts without the necessity of powering down the system is depicted in accordance with the present invention; 
     FIG. 3 depicts an exemplary table depicting possible contents and information contained within an error log in accordance with the present invention; 
     FIG. 4 depicts a flowchart illustrating an exemplary process of replacing a bad I/O adapter without powering down the system in accordance with the present invention; 
     FIG. 5 depicts a flowchart illustrating an exemplary method of removing a bad I/O adapter via hotplug in accordance with the present invention; and 
     FIG. 6 depicts an example menu allowing a user to initiate a hotplug procedure for removing and replacing a bad I/O adapter in accordance with the present invention. 
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
     With reference now to the figures, and in particular with reference to FIG. 1, a block diagram of a data processing system in which the present invention may be implemented is depicted. Data processing system  100  may be a symmetric multiprocessor (SMP) system including a plurality of processors  101 ,  102 ,  103 , and  104  connected to system bus  106 . For example, data processing system  100  may be an IBM RS/6000, a product of International Business Machines Corporation in Armonk, N.Y., implemented as a server within a network. Alternatively, a single processor system may be employed. Also connected to system bus  106  is memory controller/cache  108 , which provides an interface to a plurality of local memories  160 - 163 . I/O bus bridge  110  is connected to system bus  106  and provides an interface to I/O bus  112 . Memory controller/cache  108  and I/O bus bridge  110  may be integrated as depicted. 
     Data processing system  100  is a logically partitioned data processing system. Thus, data processing system  100  may have multiple heterogeneous operating systems (or multiple instances of a single operating system) running simultaneously. Each of theses multiple operating systems may have any number of software programs executing within in it. Data processing system  100  is logically partitioned such that different I/O adapters  120 - 121 ,  128 - 129 ,  136 - 137 , and  146 - 147  may be assigned to different logical partitions. 
     Thus, for example, suppose data processing system  100  is divided into three logical partitions, P 1 , P 2 , and P 3 . Each of I/O adapters  120 - 121 ,  128 - 129 , and  136 - 137 , each of processors  101 - 104 , and each of local memories  160 - 164  is assigned to one of the three partitions. For example, processor  101 , memory  160 , and I/O adapters  120 ,  128 , and  129  may be assigned to logical partition P 1 ; processors  102 - 103 , memory  161 , and I/O adapters  121  and  137  may be assigned to partition P 2 ; and processor  104 , memories  162 - 163 , and I/O adapters  136  and  146 - 147  may be assigned to logical partition P 3 . 
     Each operating system executing within data processing system  100  is assigned to a different logical partition. Thus, each operating system executing within data processing system  100  may access only those I/O units that are within its logical partition. Thus, for example, one instance of the Advanced Interactive Executive (AIX) operating system may be executing within partition P 1 , a second instance (image) of the AIX operating system may be executing within partition P 2 , and a Windows 2000™ operating system may be operating within logical partition P 1 . Windows 2000 is a product and trademark of Microsoft Corporation of Redmond, WA. 
     Peripheral component interconnect (PCI) Host bridge  114  connected to I/O bus  112  provides an interface to primary PCI local bus  115 . A number of Input/Output adapters  120 - 121  may be connected to primary PCI bus  115  via respective secondary PCI buses  118 - 119  and external address strobe (EADS)  116 . Typical PCI bus implementations will support between four and eight I/O adapters (i.e. expansion slots for add-in connectors). Each I/O Adapter  120 - 121  provides an interface between data processing system  100  and input/output devices such as, for example, other network computers, which are clients to data processing system  100 . 
     EADS  116 ,  124 ,  132 , and  142  are multifunction PCI-PCI bridges that support hot plugging of I/O adapters on the secondary buses  118 - 119 ,  126 - 127 ,  134 - 135 , and  144 - 145 . Hot plugging allows the addition, removal, and replacement of I/O adapters  120 - 121 ,  128 - 129 ,  136 - 137 , and  148 - 149  during runtime. Each adapter  120 - 121 ,  128 - 129 ,  136 - 137 , and  148 - 149  has its own secondary PCI bus  118 - 119 ,  126 - 127 ,  134 - 135 , and  144 - 145 , which makes hot plugging possible. Each EADS  116 ,  124 ,  132 , and  142  may be configured to function as a bridge from a primary bus  115 ,  123 ,  131 , and  141  to up to eight secondary hot plug PCI busses  118 - 119 ,  126 - 127 ,  134 - 135 , and  144 - 145 . In the depicted embodiment primary PCI buses  115 ,  123 ,  131 , and  141  are 64-bits wide, 3.3 volt (V) tolerant and run at between 16-66 MHz. Secondary PCI buses  118 - 119 ,  126 - 127 ,  134 - 135 , and  144 - 145  are 32-bits wide, wherein each pair of secondary PCI buses  118 - 119 ,  126 - 127 ,  134 - 135 , and  144 - 145  may be combined into a 64-bit wide bus. 
     Additional PCI host bridges  122  and  130  provide interfaces for additional primary PCI buses  123  and  131 . Each of additional primary PCI buses  123  and  131  are connected to a respective one of EADS  124  and  132 . Each of EADS  124  and  132  is connected to a plurality of secondary PCI buses  126 - 127  and  134 - 135  as depicted in FIG.  1 . Secondary PCI buses  126 - 127  and  134 - 135  provide a connection between a plurality of PCI I/O adapters  128 - 129  and  136 - 137  to EADS  124  and  132 . Thus, additional I/O devices, such as, for example, modems or network adapters may be supported through each of PCI I/O adapters  128 - 129  and  136 - 137 . In this manner, data processing system  100  allows connections to multiple network computers. 
     A memory mapped graphics adapter  148  and hard disk  150  may also be connected to I/O bus  112  via EADS  142  and PCI Host Bridge  140 . EADS  142  is connected to PCI Host Bridge  140  through primary PCI bus  141 . Graphics adapter  148  is connected to EADS  142  through secondary PCI bus  144  and hard disk adapter  149 , through which hard disk  150  is connected to data processing system  100 , is connected to EADS  142  through secondary PCI bus  145 . Hard disk  150  may be logically partitioned between various partitions without the need for additional hard disks. However, additional hard disks may be utilized if desired. 
     Those of ordinary skill in the art will appreciate that the hardware depicted in FIG. 1 may vary. For example, other peripheral devices, such as optical disk drives and the like, also may be used in addition to or in place of the hardware depicted. Furthermore, the present invention is not limited to implementation on a multiprocessor logically partitioned system, but may also be implemented on other types of data processing systems, such as, for example, a single processor system running a single image of an operating system, such as a typical personal computer. The depicted example is not meant to imply architectural limitations with respect to the present invention. 
     With reference now to FIG. 2, a block diagram of a system for handling machine check interrupts without the necessity of powering down the system is depicted in accordance with the present invention. A machine check interrupt is an interrupt that operating systems, such as, for example, AIX, use when the data processing system detects that an adapter has failed. A catastrophic error that will always have AIX report a machine check is a system error (SERR). A SERR may be caused for various reasons such as, for example, parity errors. Parity errors may include data parity error during a special cycle and address parity errors. A SERR may also be cause by other critical error other than parity, such as, for example, a master abort error. In prior art systems, a machine check interrupt resulted in a system shutdown. However, the present invention, provides a mechanism for handling the machine check error and replacing the bad I/O adapter without resort to shutting down the data processing system. 
     An operating system (OS)  202  executing within a data processing system, such as, for example, data processing system  100  in FIG. 1, includes an OS interrupt handler  204  for handling machine check interrupts occurring within the data processing system. OS  202  may be, for example, AIX. A run-time abstraction service (RTAS)  208  provides a mechanism for receiving machine check interrupts from PCI Host Bridges  210 - 212 . In the present example, RTAS  208  is implemented within firmware. Firmware is software stored in a memory chip that holds its content without electrical power, such as, for example, read-only memory (ROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), and non-volatile random access memory (non-volatile RAM). 
     The machine check interrupt indicates that one of I/O adapters  214 - 220 , connected to PCI Host Bridges  210 - 212  is bad (i.e. has ceased to perform correctly). OS handler  204  makes calls to RTAS  208  to determine whether a machine check has occurred and RTAS  208  presents the machine check interrupt data to OS handler  204  in the form of an error log entry. An example of an error log entry is: 
     BFE4C025 0607120300 P H sysplanar0 MACHINE —CHECK   —CHRP    
     Such an error log entry indicates to OS  202  that machine check interrupt has been received indicating that an I/O adapter  222 - 228  identified by the ID “BFE4C025” has failed. OS handler  204  than posts data to error log  206  based on the information received from RTAS  208 . Error log  300  depicted in FIG. 3 is an exemplary table, in accordance with the present invention, depicting possible contents and information contained within error log  206 . 
     The contents of error log  206  indicates which of I/O. adapters  214 - 220  is bad, if RTAS  208  is able to determine such information. Otherwise, the entry in error log  206  merely reflects the fact that a machine check interrupt has occurred within the data processing system without indicating which of I/O Adapters  214 - 220  caused the machine check interrupt. 
     Once OS handler  204  has written to error log  206 , OS  202  analyzes error log  206  to determine the identity of the failing I/O adapter  222 - 228 . For example, an AIX operating system may analyze error log  206  using the “DIAG” command and produce the following result: 
     
       
         
           
               
               
               
               
             
               
                   
                   
               
             
            
               
                   
                 A03-030: 
                 I/O bus time-out, 
                 access, or other error 
               
               
                   
                 n/a 
                 FRU:n/a 
                 U0.1-P1-I3 
               
               
                   
                   
               
            
           
         
       
     
     The data structure of the form “UO.X” indicates by “X” the drawer number of the bad adapter. Thus, in the example of above, “U0.1”(X=1), indicates that the drawer number of the bad adapter is “1.” The data structure of the form “IY” indicates the slot number of the bad adapter where “Y”, is the slot number. Thus, in the example above, the bad adapter is in slot  3 , drawer  1 . 
     If OS interrupt handler  204  is unable to determine the one of I/O adapters  214 - 220  that failed, then the data processing system is powered off. If, however, OS interrupt handler  204  is able to determine the one of I/O adapters  213 - 220  that failed, then operating system  202  will call RTAS  208  hot plug to disable the failing I/O adapter  214 - 220  and deallocate any processes bound to the failing one of I/O adapters  214 - 220 . RTAS writes directly to the EADS register of the one of EADS  214  and  218  corresponding to the bad one of I/O adapters  222 - 228  to turn off the power to the slot containing the bad one of I/O adapters  222 - 228 . 
     Once the failing one of I/O adapters  214 - 220  is disabled, an urgent sysplanar message can be sent to a user notifying the user of the bad adapter, such as, for example, via a page, a message to a video display terminal, and/or blinking lights on the slot containing the bad adapter. An urgent sysplanar is a message that the system will output to an output device, typically at predefined time intervals, so that a user may be alerted the adapter is no longer working. 
     The user will then be able to replace the bad adapter, re-enable the adapter, using the hotplug feature provided by OS  202 , RTAS  208 , and EADS  214  and  218  working in concert. The user may then reconfigure the adapter once replaced. 
     Those of ordinary skill in the art will appreciate that the hardware and software depicted in FIG. 2 may vary. For example, more or fewer I/O adapters may be used than those depicted in FIG.  2 . The depicted example is not meant to imply architectural limitations with respect to the present invention. 
     With reference now to FIG. 4, a flowchart illustrating an exemplary process of isolating a bad I/O adapter without powering down the system (i.e. a hotplug operation) is depicted in accordance with the present invention. Once the system is running and a machine check occurs indicating a bad I/O adapter, an AIX handler takes over control of the system (step  402 ). The AIX handler may be implemented as OS handler  204  in FIG.  2 . The AIX handler retrieves the error log to which the RTAS entered the machine check (step  404 ) and analyzes the error log to determine the failing adapter (step  406 ). The AIX handler then determines whether the failing card has been isolated (step  408 ). 
     If the failing card has not been isolated by the RTAS, then the AIX handler causes the system to be shut down (step  410 ). If the failing card has been isolated and determined, then disable the failing card by, for example, a call to RTAS-hot plug (step  412 ). The AIX also deallocates any processes bound to the failing I/O adapter (step  414 ). Once the failing I/O adapter has been successfully disabled, signal the appropriate expansion slot with, for example, blinking lights to indicate which I/O adapter has failed (step  416 ). The AIX then posts a message informing a user to replace the bad adapter (step  418 ). The message may be posted, for example, to a video display terminal or by page to an appropriate person assigned to maintaining the system. 
     With reference now to FIG. 5, a flowchart illustrating an exemplary method of removing a bad I/O adapter via hotplug is depicted in accordance with the present invention. Once an adapter has been identified as bad, the RTAS, such as, for example, RTAS  208  in FIG. 2, sets the visual indicator of the expansion slot in which the bad I/O adapter is located to alert a user that the expansion slot has been identified as containing a bad I/O adapter (step  502 ). The visual indicator may be, for example, blinking lights on the expansion slot. The OS, such as OS  202  in FIG. 2, unconfigures the devices associated with the bad I/O adapter (step  504 ). This process may be performed, for example, by using the AIX command “rmdev”. 
     Once the devices associated with the bad I/O adapter have been unconfigured, all nodes corresponding to devices associated with the bad I/O adapter are removed from the OS copy of the open firmware device tree provided by RTAS (step  506 ). RTAS then isolates and turns the power off to the expansion slot containing the bad I/O adapter (step  508 ). RTAS then sets the affected expansion slot&#39;s visual indicator to a condition that informs a user that the I/O adapter connected to that expansion slot may be removed (step  510 ). Once, the bad I/O adapter has been removed, the RTAS turns off the visual indicator to the affected expansion slot (step  512 ) indicating that the user has removed the correct I/O adapter. The user may then insert a replacement I/O adapter and reconfigure the system to utilize the new I/O adapter. Thus, a bad I/O adapter may be removed and replaced during runtime (i.e. without shutting down the system). 
     Once the user has been notified of the bad I/O adapter, the process of removing and replacing the bad I/O adapter may be initiated through a menu, such as, for example, an AIX System Management Interface Tool (SMIT) menu. An example of a SMIT menu is depicted in FIG.  6 . Menu  600  provides a user with a list of hotplug services performed by the data processing system, such as data processing system  100  in FIG.  1 . In menu  600 , a user may select item  3  to place the bad I/O adapter in a condition in which the user can remove and replace the bad I/O adapter within the data processing system. 
     It is important to note that while the present invention has been described in the context of a fully functioning data processing system, those of ordinary skill in the art will appreciate that the processes of the present invention are capable of being distributed in the form of a computer readable medium of instructions and a variety of forms and that the present invention applies equally regardless of the particular type of signal bearing media actually used to carry out the distribution. Examples of computer readable media include recordable-type media such a floppy disc, a hard disk drive, a RAM, and CD-ROMs and transmission-type media such as digital and analog communications links. 
     The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiment was chosen and described in order to best explain the principles of the invention, the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.